[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:04:12 dist-test-slave-23bp krb5kdc[1761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:04:12 dist-test-slave-23bp krb5kdc[1761](info): set up 2 sockets
Sep 28 02:04:12 dist-test-slave-23bp krb5kdc[1761](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 02:04:13.865969 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:04:13 dist-test-slave-23bp krb5kdc[1761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:04:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:04:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:04:14.041186 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:40615
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:34107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:40615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:14.361301 1774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:14.361639 1774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:14.361819 1774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:14.368975 1774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:14.369141 1774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:14.369228 1774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:14.369313 1774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:14.369382 1774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:14.379299 1774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:34107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:40615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:40615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:14.380661 1774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:14.381981 1774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:14.393522 1779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:14.393554 1782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:14.393590 1780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:14.394810 1774 server_base.cc:1034] running on GCE node
I20260928 02:04:14.395700 1774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:14.397079 1774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:14.398331 1774 hybrid_clock.cc:648] HybridClock initialized: now 1790561054398281 us; error 55 us; skew 500 ppm
I20260928 02:04:14.398716 1774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:14 dist-test-slave-23bp krb5kdc[1761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561054, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:14.401865 1774 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:04:14.403388 1774 webserver.cc:460] Webserver started at http://127.1.182.190:43459/ using document root <none> and password file <none>
I20260928 02:04:14.404068 1774 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:14.404214 1774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:14.404515 1774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:14.407501 1774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "64d388eefb28485aac2c955bef582fe9"
format_stamp: "Formatted at 2026-09-28 02:04:14 on dist-test-slave-23bp"
server_key: "15906e7106fa56e52e49edc56c2634cb"
server_key_iv: "3ed6eb6de9e4266efd48370f7b0011d7"
server_key_version: "encryptionkey@0"
I20260928 02:04:14.408260 1774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "64d388eefb28485aac2c955bef582fe9"
format_stamp: "Formatted at 2026-09-28 02:04:14 on dist-test-slave-23bp"
server_key: "15906e7106fa56e52e49edc56c2634cb"
server_key_iv: "3ed6eb6de9e4266efd48370f7b0011d7"
server_key_version: "encryptionkey@0"
I20260928 02:04:14.413353 1774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:04:14.416771 1789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:14.418254 1774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:04:14.418416 1774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "64d388eefb28485aac2c955bef582fe9"
format_stamp: "Formatted at 2026-09-28 02:04:14 on dist-test-slave-23bp"
server_key: "15906e7106fa56e52e49edc56c2634cb"
server_key_iv: "3ed6eb6de9e4266efd48370f7b0011d7"
server_key_version: "encryptionkey@0"
I20260928 02:04:14.418629 1774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:14.438099 1774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:14.442565 1774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:14.442860 1774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:14.462682 1774 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:40615
I20260928 02:04:14.462687 1840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:40615 every 8 connection(s)
I20260928 02:04:14.464195 1774 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:04:14.466732 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 1774
I20260928 02:04:14.467042 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:04:14.468120 1841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:14.469020 1754 external_mini_cluster.cc:1428] Setting key 3fba445b2cd07ccf0463c7ef460c1ee1
I20260928 02:04:14.478359 1841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9: Bootstrap starting.
I20260928 02:04:14.481233 1841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:14.482174 1841 log.cc:824] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:14.485680 1841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9: No bootstrap required, opened a new log
I20260928 02:04:14.494180 1841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d388eefb28485aac2c955bef582fe9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 40615 } }
I20260928 02:04:14.494534 1841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:14.494675 1841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d388eefb28485aac2c955bef582fe9, State: Initialized, Role: FOLLOWER
I20260928 02:04:14.495360 1841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d388eefb28485aac2c955bef582fe9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 40615 } }
I20260928 02:04:14.495667 1841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:04:14.495839 1841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:04:14.496021 1841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:14.498140 1841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d388eefb28485aac2c955bef582fe9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 40615 } }
I20260928 02:04:14.498714 1841 leader_election.cc:302] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64d388eefb28485aac2c955bef582fe9; no voters:
Sep 28 02:04:14 dist-test-slave-23bp krb5kdc[1761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:14.499414 1841 leader_election.cc:288] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:04:14.499559 1846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:14.501155 1846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9 [term 1 LEADER]: Becoming Leader. State: Replica: 64d388eefb28485aac2c955bef582fe9, State: Running, Role: LEADER
I20260928 02:04:14.501721 1841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:14.501845 1846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d388eefb28485aac2c955bef582fe9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 40615 } }
I20260928 02:04:14.504443 1848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64d388eefb28485aac2c955bef582fe9. Latest consensus state: current_term: 1 leader_uuid: "64d388eefb28485aac2c955bef582fe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d388eefb28485aac2c955bef582fe9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 40615 } } }
I20260928 02:04:14.504873 1848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:14.505399 1847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64d388eefb28485aac2c955bef582fe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d388eefb28485aac2c955bef582fe9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 40615 } } }
I20260928 02:04:14.505650 1847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:14.509341 1851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:14.514343 1843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:14.472790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40444 (local address 127.1.182.190:40615)
0928 02:04:14.473678 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:14.473692 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:14.475015 (+ 1323us) server_negotiation.cc:407] Connection header received
0928 02:04:14.484768 (+ 9753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:14.484804 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:14.485281 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:14.485700 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:14.488821 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:14.490922 (+ 2101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:14.492752 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:14.493786 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:14.493973 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:14.501026 (+ 7053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:14.501037 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:14.501060 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:14.501127 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:14.506313 (+ 5186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:14.507737 (+ 1424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:14.507750 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:14.507762 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:14.508030 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:14.508757 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:14.508765 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:14.508771 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:14.509645 (+ 874us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:14.511829 (+ 2184us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:14.512187 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 02:04:14.512769 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":396,"threads_started":1}
I20260928 02:04:14.516495 1851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:14.531013 1851 catalog_manager.cc:1347] Generated new cluster ID: ffd6a420807e43eb96f1443fd2c28147
I20260928 02:04:14.531616 1851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:14.567956 1851 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:04:14.569576 1851 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:14.578984 1851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64d388eefb28485aac2c955bef582fe9: Generated new TSK 0
I20260928 02:04:14.579840 1851 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.182.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
I20260928 02:04:14.637225 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:40615
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:34107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:14.978057 1868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:14.978406 1868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:14.978593 1868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:14.986515 1868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:14.986685 1868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:14.986878 1868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:04:14.997344 1868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:34107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:40615
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:14.998739 1868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:15.000188 1868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:15.014045 1873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:15.014276 1876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:15.014356 1874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:15.015434 1868 server_base.cc:1034] running on GCE node
I20260928 02:04:15.016093 1868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:15.017097 1868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:15.018325 1868 hybrid_clock.cc:648] HybridClock initialized: now 1790561055018282 us; error 41 us; skew 500 ppm
I20260928 02:04:15.018652 1868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:15 dist-test-slave-23bp krb5kdc[1761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561055, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:15.021910 1868 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:04:15.023641 1868 webserver.cc:460] Webserver started at http://127.1.182.129:41863/ using document root <none> and password file <none>
I20260928 02:04:15.024375 1868 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:15.024523 1868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:15.024869 1868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:15.027915 1868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "9c6e3361405540a0a415f1046fb19d72"
format_stamp: "Formatted at 2026-09-28 02:04:15 on dist-test-slave-23bp"
server_key: "3d33f0989d04b42f6d43c8388fe7c3f6"
server_key_iv: "c0a3d08d0c1dc4910f0a2527830ebe0d"
server_key_version: "encryptionkey@0"
I20260928 02:04:15.028729 1868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "9c6e3361405540a0a415f1046fb19d72"
format_stamp: "Formatted at 2026-09-28 02:04:15 on dist-test-slave-23bp"
server_key: "3d33f0989d04b42f6d43c8388fe7c3f6"
server_key_iv: "c0a3d08d0c1dc4910f0a2527830ebe0d"
server_key_version: "encryptionkey@0"
I20260928 02:04:15.034279 1868 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:04:15.037878 1883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:15.039266 1868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:04:15.039434 1868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "9c6e3361405540a0a415f1046fb19d72"
format_stamp: "Formatted at 2026-09-28 02:04:15 on dist-test-slave-23bp"
server_key: "3d33f0989d04b42f6d43c8388fe7c3f6"
server_key_iv: "c0a3d08d0c1dc4910f0a2527830ebe0d"
server_key_version: "encryptionkey@0"
I20260928 02:04:15.039631 1868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:15.054736 1868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:15.058929 1868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:15.059253 1868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:15.060578 1868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:15.061893 1868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:15.061997 1868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:15.062162 1868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:15.062252 1868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:15.097378 1868 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:45113
I20260928 02:04:15.097409 1995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:45113 every 8 connection(s)
I20260928 02:04:15.098671 1868 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:04:15.106364 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 1868
I20260928 02:04:15.106607 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:04:15.107302 1754 external_mini_cluster.cc:1428] Setting key 1719dab2b72e9e054769e212a5cde9dc
Sep 28 02:04:15 dist-test-slave-23bp krb5kdc[1761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561055, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:15.120591 1998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:15.101263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:32795 (local address 127.1.182.190:40615)
0928 02:04:15.101686 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:15.101690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:15.102755 (+ 1065us) server_negotiation.cc:407] Connection header received
0928 02:04:15.104675 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:15.104680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:15.104791 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:15.104911 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:15.106399 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:15.107325 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:15.109848 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:15.110262 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:15.110374 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:15.114939 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:15.114944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:15.114947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:15.114978 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:15.117533 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:15.118283 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:15.118289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:15.118293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:15.118413 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:15.118843 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:15.118849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:15.118852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:15.119176 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:15.119324 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:15.120013 (+ 689us) server_negotiation.cc:299] Negotiation successful
0928 02:04:15.120228 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":214,"threads_started":1}
I20260928 02:04:15.121606 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:15.101903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:40615 (local address 127.1.182.129:32795)
0928 02:04:15.102584 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:15.102626 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:15.104409 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:15.105158 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:15.105166 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:15.105661 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:15.106212 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:15.106240 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:15.107518 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:15.107524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:15.109704 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:15.109719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:15.110519 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:15.110529 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:15.110875 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:15.111668 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:15.111706 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:15.114694 (+ 2988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:15.117688 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:15.117696 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:15.117715 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:15.118132 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:15.118522 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:15.118528 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:15.118531 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:15.118734 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:15.119307 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:15.119316 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:15.119658 (+ 342us) client_negotiation.cc:769] Sending connection context
0928 02:04:15.119931 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 02:04:15.120471 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":262,"threads_started":1}
I20260928 02:04:15.123399 1996 heartbeater.cc:347] Connected to a master server at 127.1.182.190:40615
I20260928 02:04:15.123744 1996 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.182.130@KRBTEST.COM; defaulting to no policy
I20260928 02:04:15.124504 1996 heartbeater.cc:511] Master 127.1.182.190:40615 requested a full tablet report, sending...
I20260928 02:04:15.126789 1806 ts_manager.cc:194] Registered new tserver with Master: 9c6e3361405540a0a415f1046fb19d72 (127.1.182.129:45113)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.130@KRBTEST.COM" created.
I20260928 02:04:15.128487 1806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:32795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
I20260928 02:04:15.151292 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:40615
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:34107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:15.491819 2002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:15.492094 2002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:15.492268 2002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:15.499908 2002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:15.500033 2002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:15.500221 2002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:04:15.510118 2002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:34107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:40615
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:15.511422 2002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:15.512768 2002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:15.525931 2010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:15.525880 2007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:15.525880 2008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:15.526922 2002 server_base.cc:1034] running on GCE node
I20260928 02:04:15.527549 2002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:15.528501 2002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:15.529742 2002 hybrid_clock.cc:648] HybridClock initialized: now 1790561055529695 us; error 60 us; skew 500 ppm
I20260928 02:04:15.530049 2002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:15 dist-test-slave-23bp krb5kdc[1761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561055, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:15.533475 2002 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:04:15.534827 2002 webserver.cc:460] Webserver started at http://127.1.182.130:33415/ using document root <none> and password file <none>
I20260928 02:04:15.535647 2002 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:15.535794 2002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:15.536152 2002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:15.538933 2002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "615aae2e9e774e2aa9e41ed62345058a"
format_stamp: "Formatted at 2026-09-28 02:04:15 on dist-test-slave-23bp"
server_key: "e6c4cd767eea70a43fa7571b4660d906"
server_key_iv: "e0be65e8ce0f8bce0fc712443548770a"
server_key_version: "encryptionkey@0"
I20260928 02:04:15.539686 2002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "615aae2e9e774e2aa9e41ed62345058a"
format_stamp: "Formatted at 2026-09-28 02:04:15 on dist-test-slave-23bp"
server_key: "e6c4cd767eea70a43fa7571b4660d906"
server_key_iv: "e0be65e8ce0f8bce0fc712443548770a"
server_key_version: "encryptionkey@0"
I20260928 02:04:15.544770 2002 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:04:15.548028 2017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:15.549290 2002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:04:15.549448 2002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "615aae2e9e774e2aa9e41ed62345058a"
format_stamp: "Formatted at 2026-09-28 02:04:15 on dist-test-slave-23bp"
server_key: "e6c4cd767eea70a43fa7571b4660d906"
server_key_iv: "e0be65e8ce0f8bce0fc712443548770a"
server_key_version: "encryptionkey@0"
I20260928 02:04:15.549635 2002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:15.584789 2002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:15.588773 2002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:15.589054 2002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:15.590041 2002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:15.591388 2002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:15.591492 2002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:15.591641 2002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:15.591719 2002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:15.627532 2002 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:44009
I20260928 02:04:15.627586 2129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:44009 every 8 connection(s)
I20260928 02:04:15.628939 2002 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:04:15.637465 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 2002
I20260928 02:04:15.637681 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:04:15.638105 1754 external_mini_cluster.cc:1428] Setting key cceee75c54c05a8e158d7d316c4af32c
Sep 28 02:04:15 dist-test-slave-23bp krb5kdc[1761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561055, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:15.650142 2132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:15.631533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:60831 (local address 127.1.182.190:40615)
0928 02:04:15.631961 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:15.631965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:15.633206 (+ 1241us) server_negotiation.cc:407] Connection header received
0928 02:04:15.634992 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:15.634998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:15.635127 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:15.635251 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:15.636731 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:15.637420 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:15.639311 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:15.639852 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:15.639955 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:15.643426 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:15.643435 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:15.643443 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:15.643489 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:15.646867 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:15.647588 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:15.647597 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:15.647603 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:15.647757 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:15.648160 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:15.648167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:15.648179 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:15.648577 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:15.648725 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:15.649384 (+ 659us) server_negotiation.cc:299] Negotiation successful
0928 02:04:15.649637 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":238,"threads_started":1}
I20260928 02:04:15.650755 2133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:15.632022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:40615 (local address 127.1.182.130:60831)
0928 02:04:15.633047 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:15.633084 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:15.634731 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:15.635491 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:15.635499 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:15.636004 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:15.636558 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:15.636574 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:15.637570 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:15.637575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:15.639170 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:15.639184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:15.640082 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:15.640090 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:15.640301 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:15.640995 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:15.641016 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:15.643176 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:15.647003 (+ 3827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:15.647009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:15.647022 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:15.647474 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:15.647865 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:15.647868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:15.647870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:15.648036 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:15.648699 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:15.648705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:15.649016 (+ 311us) client_negotiation.cc:769] Sending connection context
0928 02:04:15.649234 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 02:04:15.649597 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":646,"threads_started":1}
I20260928 02:04:15.652251 2130 heartbeater.cc:347] Connected to a master server at 127.1.182.190:40615
I20260928 02:04:15.652616 2130 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:15.653542 2130 heartbeater.cc:511] Master 127.1.182.190:40615 requested a full tablet report, sending...
I20260928 02:04:15.655162 1806 ts_manager.cc:194] Registered new tserver with Master: 615aae2e9e774e2aa9e41ed62345058a (127.1.182.130:44009)
WARNING: no policy specified for kudu/127.1.182.131@KRBTEST.COM; defaulting to no policy
I20260928 02:04:15.656020 1806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:60831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
I20260928 02:04:15.678442 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:40615
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:34107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:16.009836 2136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:16.010147 2136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:16.010253 2136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:16.017675 2136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:16.017795 2136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:16.017912 2136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:04:16.027958 2136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:34107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:40615
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:16.029414 2136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:16.031169 2136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:16.044514 2144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:16.044534 2142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:16.044775 2141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:16.045295 2136 server_base.cc:1034] running on GCE node
I20260928 02:04:16.045805 2136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:16.046624 2136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:16.047812 2136 hybrid_clock.cc:648] HybridClock initialized: now 1790561056047779 us; error 24 us; skew 500 ppm
I20260928 02:04:16.048070 2136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:16 dist-test-slave-23bp krb5kdc[1761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561056, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:16.051625 2136 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:04:16.053395 2136 webserver.cc:460] Webserver started at http://127.1.182.131:34931/ using document root <none> and password file <none>
I20260928 02:04:16.054250 2136 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:16.054394 2136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:16.054754 2136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:16.058182 2136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "e99dd52556424b4887fd4fec2b8bae28"
format_stamp: "Formatted at 2026-09-28 02:04:16 on dist-test-slave-23bp"
server_key: "3c657fa7acfca80e1faec4665de9b949"
server_key_iv: "c19e119549e94c87f30b462680f9cd40"
server_key_version: "encryptionkey@0"
I20260928 02:04:16.059145 2136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "e99dd52556424b4887fd4fec2b8bae28"
format_stamp: "Formatted at 2026-09-28 02:04:16 on dist-test-slave-23bp"
server_key: "3c657fa7acfca80e1faec4665de9b949"
server_key_iv: "c19e119549e94c87f30b462680f9cd40"
server_key_version: "encryptionkey@0"
I20260928 02:04:16.064555 2136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:04:16.068240 2151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:16.069738 2136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:04:16.069903 2136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "e99dd52556424b4887fd4fec2b8bae28"
format_stamp: "Formatted at 2026-09-28 02:04:16 on dist-test-slave-23bp"
server_key: "3c657fa7acfca80e1faec4665de9b949"
server_key_iv: "c19e119549e94c87f30b462680f9cd40"
server_key_version: "encryptionkey@0"
I20260928 02:04:16.070076 2136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:16.089783 2136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:16.093670 2136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:16.093941 2136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:16.095028 2136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:16.096629 2136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:16.096697 2136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:16.096786 2136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:16.096822 2136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:16.131820 1996 heartbeater.cc:503] Master 127.1.182.190:40615 was elected leader, sending a full tablet report...
I20260928 02:04:16.134086 2136 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:41253
I20260928 02:04:16.134178 2263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:41253 every 8 connection(s)
I20260928 02:04:16.135499 2136 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:04:16.145385 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 2136
I20260928 02:04:16.145617 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:04:16.146085 1754 external_mini_cluster.cc:1428] Setting key 164f558d86d682243584ee4c77c39363
Sep 28 02:04:16 dist-test-slave-23bp krb5kdc[1761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561056, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:16.156016 2132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:16.138161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:55145 (local address 127.1.182.190:40615)
0928 02:04:16.138327 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:16.138331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:16.139497 (+ 1166us) server_negotiation.cc:407] Connection header received
0928 02:04:16.141225 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:16.141229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:16.141296 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:16.141385 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:16.142850 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:16.143713 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:16.145517 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:16.145946 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:16.146042 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:16.150035 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:16.150041 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:16.150044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:16.150097 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:16.152647 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:16.153329 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:16.153337 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:16.153342 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:16.153869 (+ 527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:16.154295 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:16.154301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:16.154304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:16.154577 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:16.154741 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:16.155468 (+ 727us) server_negotiation.cc:299] Negotiation successful
0928 02:04:16.155697 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:04:16.157023 2266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:16.138622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:40615 (local address 127.1.182.131:55145)
0928 02:04:16.139343 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:16.139386 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:16.140976 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:16.141584 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:16.141592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:16.142108 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:16.142701 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:16.142719 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:16.143902 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:16.143905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:16.145326 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:16.145342 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:16.146172 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:16.146181 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:16.146472 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:16.147358 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:16.147392 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:16.149820 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:16.152786 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:16.152799 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:16.152815 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:16.153211 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:16.153992 (+ 781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:16.153997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:16.154001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:16.154182 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:16.154704 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:16.154714 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:16.155053 (+ 339us) client_negotiation.cc:769] Sending connection context
0928 02:04:16.155385 (+ 332us) client_negotiation.cc:240] Negotiation successful
0928 02:04:16.155909 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":259,"threads_started":1}
I20260928 02:04:16.158443 2264 heartbeater.cc:347] Connected to a master server at 127.1.182.190:40615
I20260928 02:04:16.158764 2264 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:16.159550 2264 heartbeater.cc:511] Master 127.1.182.190:40615 requested a full tablet report, sending...
I20260928 02:04:16.161145 1806 ts_manager.cc:194] Registered new tserver with Master: e99dd52556424b4887fd4fec2b8bae28 (127.1.182.131:41253)
I20260928 02:04:16.162139 1806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:55145
I20260928 02:04:16.162529 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:04:16 dist-test-slave-23bp krb5kdc[1761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561056, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:04:16 dist-test-slave-23bp krb5kdc[1761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561056, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:04:16.208714 2271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:16.192762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.1.182.129:45113)
0928 02:04:16.193617 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:16.193624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:16.193653 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:04:16.193711 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:16.193718 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:16.193940 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:16.194079 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:16.195730 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:16.196888 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:16.198197 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:16.198779 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:16.198886 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:16.203485 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:16.203498 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:16.203514 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:16.203549 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:16.206275 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:16.206862 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:16.206870 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:16.206872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:16.206989 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:16.207386 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:16.207389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:16.207391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:16.207728 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:16.207945 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:16.208185 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 02:04:16.208387 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":353,"threads_started":1}
W20260928 02:04:16.210083 1911 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56486
Sep 28 02:04:16 dist-test-slave-23bp krb5kdc[1761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:04:16 dist-test-slave-23bp krb5kdc[1761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:04:16.253764 2271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:16.238794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.1.182.129:45113)
0928 02:04:16.239000 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:16.239009 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:16.239457 (+ 448us) server_negotiation.cc:407] Connection header received
0928 02:04:16.239675 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:16.239680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:16.239751 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:16.239853 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:16.241188 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:16.242132 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:16.243699 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:16.244131 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:16.244260 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:16.248494 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:16.248499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:16.248502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:16.248534 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:16.251030 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:16.251918 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:16.251923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:16.251925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:16.252039 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:16.252399 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:16.252403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:16.252404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:16.252631 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:16.252884 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:16.253219 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 02:04:16.253435 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:04:16.255234 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 1868
I20260928 02:04:16.267915 2271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:16.256226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56500 (local address 127.1.182.129:45113)
0928 02:04:16.256427 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:16.256433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:16.256698 (+ 265us) server_negotiation.cc:407] Connection header received
0928 02:04:16.256883 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:16.256892 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:16.256945 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:16.257038 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:16.258180 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:16.258843 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:16.260150 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:16.260570 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:16.260705 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:16.262853 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:16.262858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:16.262861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:16.262887 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:16.265415 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:16.266039 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:16.266046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:16.266049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:16.266175 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:16.266530 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:16.266537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:16.266542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:16.266803 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:16.267041 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:16.267373 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 02:04:16.267562 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:04:16.268728 1991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:16.353295 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 1868
I20260928 02:04:16.370309 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 2002
Sep 28 02:04:16 dist-test-slave-23bp krb5kdc[1761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.130@KRBTEST.COM
I20260928 02:04:16.385288 2278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:16.371219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37186 (local address 127.1.182.130:44009)
0928 02:04:16.371747 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:16.371755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:16.371780 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:04:16.371838 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:16.371844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:16.372051 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:16.372180 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:16.373256 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:16.374238 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:16.375673 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:16.376210 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:16.376308 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:16.380125 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:16.380136 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:16.380151 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:16.380184 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:16.382805 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:16.383445 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:16.383451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:16.383453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:16.383567 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:16.383906 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:16.383909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:16.383911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:16.384222 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:16.384437 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:16.384703 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 02:04:16.384925 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":271,"threads_started":1}
I20260928 02:04:16.386494 2125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:16.473631 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 2002
I20260928 02:04:16.495776 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 2136
Sep 28 02:04:16 dist-test-slave-23bp krb5kdc[1761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.131@KRBTEST.COM
I20260928 02:04:16.512753 2280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:16.496795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36896 (local address 127.1.182.131:41253)
0928 02:04:16.497524 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:16.497536 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:16.497625 (+ 89us) server_negotiation.cc:407] Connection header received
0928 02:04:16.497694 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:16.497701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:16.498176 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:16.498371 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:16.499683 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:16.500685 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:16.501970 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:16.502708 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:16.502849 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:16.507266 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:16.507277 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:16.507292 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:16.507324 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:16.510083 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:16.510702 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:16.510709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:16.510711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:16.510831 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:16.511345 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:16.511348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:16.511350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:16.511671 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:16.511899 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:16.512266 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 02:04:16.512451 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":364,"threads_started":1}
I20260928 02:04:16.514382 2259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:16.595782 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 2136
I20260928 02:04:16.612860 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 1774
I20260928 02:04:16.614023 1836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:16.715610 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 1774
2026-09-28T02:04:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3951 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:04:16 dist-test-slave-23bp krb5kdc[2285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:04:16 dist-test-slave-23bp krb5kdc[2285](info): set up 2 sockets
Sep 28 02:04:16 dist-test-slave-23bp krb5kdc[2285](info): commencing operation
krb5kdc: starting...
W20260928 02:04:17.825944 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:04:17 dist-test-slave-23bp krb5kdc[2285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:04:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:04:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:04:18.007764 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:40465
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:40465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:18.332398 2298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:18.332686 2298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:18.332775 2298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:18.339974 2298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:18.340075 2298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:18.340119 2298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:18.340167 2298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:18.340204 2298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:18.349862 2298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:40465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:40465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:18.351190 2298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:18.352444 2298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:18.363909 2304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:18.364028 2306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:18.364087 2303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:18.365272 2298 server_base.cc:1034] running on GCE node
I20260928 02:04:18.365949 2298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:18.367343 2298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:18.368573 2298 hybrid_clock.cc:648] HybridClock initialized: now 1790561058368535 us; error 44 us; skew 500 ppm
I20260928 02:04:18.368911 2298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:18 dist-test-slave-23bp krb5kdc[2285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561058, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:18.378713 2298 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:04:18.380283 2298 webserver.cc:460] Webserver started at http://127.1.182.190:35207/ using document root <none> and password file <none>
I20260928 02:04:18.380940 2298 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:18.381035 2298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:18.381300 2298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:18.384245 2298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "a424bdfe82bb44d7936a5261804d6f78"
format_stamp: "Formatted at 2026-09-28 02:04:18 on dist-test-slave-23bp"
server_key: "44cc5a91b37fb9d17c5413d4c4f9bed3"
server_key_iv: "faa86ad17bd2367aa1cea65b6624b365"
server_key_version: "encryptionkey@0"
I20260928 02:04:18.384967 2298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "a424bdfe82bb44d7936a5261804d6f78"
format_stamp: "Formatted at 2026-09-28 02:04:18 on dist-test-slave-23bp"
server_key: "44cc5a91b37fb9d17c5413d4c4f9bed3"
server_key_iv: "faa86ad17bd2367aa1cea65b6624b365"
server_key_version: "encryptionkey@0"
I20260928 02:04:18.389883 2298 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:04:18.393144 2313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:18.394431 2298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:04:18.394601 2298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "a424bdfe82bb44d7936a5261804d6f78"
format_stamp: "Formatted at 2026-09-28 02:04:18 on dist-test-slave-23bp"
server_key: "44cc5a91b37fb9d17c5413d4c4f9bed3"
server_key_iv: "faa86ad17bd2367aa1cea65b6624b365"
server_key_version: "encryptionkey@0"
I20260928 02:04:18.394776 2298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:18.411710 2298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:18.416178 2298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:18.416404 2298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:18.433666 2298 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:40465
I20260928 02:04:18.433689 2364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:40465 every 8 connection(s)
I20260928 02:04:18.435039 2298 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:04:18.436261 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 2298
I20260928 02:04:18.436497 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:04:18.437166 1754 external_mini_cluster.cc:1428] Setting key 6ee670bb995593fb567e39feeed394f9
I20260928 02:04:18.439635 2365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 02:04:18 dist-test-slave-23bp krb5kdc[2285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:18.452664 2365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78: Bootstrap starting.
I20260928 02:04:18.456773 2365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:18.458000 2365 log.cc:824] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:18.461273 2365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78: No bootstrap required, opened a new log
I20260928 02:04:18.461678 2368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:18.439306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34280 (local address 127.1.182.190:40465)
0928 02:04:18.440113 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:18.440127 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:18.440178 (+ 51us) server_negotiation.cc:407] Connection header received
0928 02:04:18.442489 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:18.442515 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:18.443031 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:18.443492 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:18.444793 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:18.446807 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:18.448093 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:18.449093 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:18.449281 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:18.453433 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:18.453443 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:18.453467 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:18.453508 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:18.457450 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:18.458074 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:18.458084 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:18.458093 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:18.458274 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:18.458662 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:18.458670 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:18.458675 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:18.459333 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:18.459624 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:18.460005 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 02:04:18.460401 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":377,"threads_started":1}
I20260928 02:04:18.469501 2365 raft_consensus.cc:348] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a424bdfe82bb44d7936a5261804d6f78" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 40465 } }
I20260928 02:04:18.469839 2365 raft_consensus.cc:374] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:18.469913 2365 raft_consensus.cc:729] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a424bdfe82bb44d7936a5261804d6f78, State: Initialized, Role: FOLLOWER
I20260928 02:04:18.470438 2365 consensus_queue.cc:260] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a424bdfe82bb44d7936a5261804d6f78" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 40465 } }
I20260928 02:04:18.470628 2365 raft_consensus.cc:388] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:04:18.470721 2365 raft_consensus.cc:482] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:04:18.470846 2365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:18.473102 2365 raft_consensus.cc:504] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a424bdfe82bb44d7936a5261804d6f78" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 40465 } }
I20260928 02:04:18.473513 2365 leader_election.cc:302] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a424bdfe82bb44d7936a5261804d6f78; no voters:
I20260928 02:04:18.474001 2365 leader_election.cc:288] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:04:18.474148 2370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:18.474798 2370 raft_consensus.cc:686] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78 [term 1 LEADER]: Becoming Leader. State: Replica: a424bdfe82bb44d7936a5261804d6f78, State: Running, Role: LEADER
I20260928 02:04:18.475385 2370 consensus_queue.cc:237] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a424bdfe82bb44d7936a5261804d6f78" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 40465 } }
I20260928 02:04:18.476069 2365 sys_catalog.cc:564] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:18.478030 2371 sys_catalog.cc:455] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a424bdfe82bb44d7936a5261804d6f78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a424bdfe82bb44d7936a5261804d6f78" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 40465 } } }
I20260928 02:04:18.478403 2371 sys_catalog.cc:458] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:18.478019 2372 sys_catalog.cc:455] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a424bdfe82bb44d7936a5261804d6f78. Latest consensus state: current_term: 1 leader_uuid: "a424bdfe82bb44d7936a5261804d6f78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a424bdfe82bb44d7936a5261804d6f78" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 40465 } } }
I20260928 02:04:18.478881 2372 sys_catalog.cc:458] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:18.480207 2375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:18.485983 2375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:18.495306 2375 catalog_manager.cc:1347] Generated new cluster ID: ff06ff44dec144d793d4632564593e95
I20260928 02:04:18.495509 2375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:18.514638 2375 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:04:18.516201 2375 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:18.523777 2375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a424bdfe82bb44d7936a5261804d6f78: Generated new TSK 0
I20260928 02:04:18.524807 2375 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.182.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
I20260928 02:04:18.579167 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:40465
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:18.910741 2391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:18.911023 2391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:18.911195 2391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:18.918460 2391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:18.918582 2391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:18.918699 2391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:04:18.928512 2391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:40465
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:18.929781 2391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:18.931088 2391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:18.944171 2396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:18.944125 2399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:18.944125 2397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:18.945329 2391 server_base.cc:1034] running on GCE node
I20260928 02:04:18.945926 2391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:18.946920 2391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:18.948120 2391 hybrid_clock.cc:648] HybridClock initialized: now 1790561058948071 us; error 74 us; skew 500 ppm
I20260928 02:04:18.948426 2391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:18 dist-test-slave-23bp krb5kdc[2285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561058, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:18.951520 2391 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:04:18.953048 2391 webserver.cc:460] Webserver started at http://127.1.182.129:40589/ using document root <none> and password file <none>
I20260928 02:04:18.953737 2391 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:18.953881 2391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:18.954188 2391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:18.957206 2391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "527b257fbfc140f0b93301646c0635dc"
format_stamp: "Formatted at 2026-09-28 02:04:18 on dist-test-slave-23bp"
server_key: "c017303c485d8d0771194070a248b598"
server_key_iv: "c8b630acb9d32ec35c1117c14eb3df36"
server_key_version: "encryptionkey@0"
I20260928 02:04:18.958020 2391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "527b257fbfc140f0b93301646c0635dc"
format_stamp: "Formatted at 2026-09-28 02:04:18 on dist-test-slave-23bp"
server_key: "c017303c485d8d0771194070a248b598"
server_key_iv: "c8b630acb9d32ec35c1117c14eb3df36"
server_key_version: "encryptionkey@0"
I20260928 02:04:18.963029 2391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:04:18.966663 2406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:18.968125 2391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:04:18.968317 2391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "527b257fbfc140f0b93301646c0635dc"
format_stamp: "Formatted at 2026-09-28 02:04:18 on dist-test-slave-23bp"
server_key: "c017303c485d8d0771194070a248b598"
server_key_iv: "c8b630acb9d32ec35c1117c14eb3df36"
server_key_version: "encryptionkey@0"
I20260928 02:04:18.968534 2391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:18.989375 2391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:18.993559 2391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:18.993865 2391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:18.994822 2391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:18.996181 2391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:18.996289 2391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:18.996429 2391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:18.996500 2391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:19.029328 2391 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:41945
I20260928 02:04:19.029383 2518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:41945 every 8 connection(s)
I20260928 02:04:19.030685 2391 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:04:19.038359 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 2391
I20260928 02:04:19.038582 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:04:19.039276 1754 external_mini_cluster.cc:1428] Setting key ea3d1a166277a72d5b336a5a88629fb2
Sep 28 02:04:19 dist-test-slave-23bp krb5kdc[2285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561058, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:19.052549 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:19.033338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:59813 (local address 127.1.182.190:40465)
0928 02:04:19.033719 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:19.033723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:19.034765 (+ 1042us) server_negotiation.cc:407] Connection header received
0928 02:04:19.036662 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:19.036668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:19.036763 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:19.036883 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:19.038884 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:19.039714 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:19.041501 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:19.041889 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:19.041985 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:19.045809 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:19.045815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:19.045821 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:19.045864 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:19.049380 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:19.050052 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:19.050060 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:19.050066 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:19.050208 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:19.050592 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:19.050599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:19.050605 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:19.050960 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:19.051149 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:19.051830 (+ 681us) server_negotiation.cc:299] Negotiation successful
0928 02:04:19.052083 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":188,"threads_started":1}
I20260928 02:04:19.053316 2522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:19.033905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:40465 (local address 127.1.182.129:59813)
0928 02:04:19.034598 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:19.034636 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:19.036397 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:19.037158 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:19.037171 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:19.037818 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:19.038664 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:19.038688 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:19.039901 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:19.039905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:19.041362 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:19.041376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:19.042104 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:19.042112 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:19.042466 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:19.043255 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:19.043277 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:19.045583 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:19.049518 (+ 3935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:19.049525 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:19.049538 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:19.049933 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:19.050315 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:19.050318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:19.050321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:19.050489 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:19.051138 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:19.051143 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:19.051461 (+ 318us) client_negotiation.cc:769] Sending connection context
0928 02:04:19.051695 (+ 234us) client_negotiation.cc:240] Negotiation successful
0928 02:04:19.052082 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":261,"threads_started":1}
I20260928 02:04:19.054859 2519 heartbeater.cc:347] Connected to a master server at 127.1.182.190:40465
I20260928 02:04:19.055245 2519 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:19.055960 2519 heartbeater.cc:511] Master 127.1.182.190:40465 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.182.130@KRBTEST.COM; defaulting to no policy
I20260928 02:04:19.058211 2330 ts_manager.cc:194] Registered new tserver with Master: 527b257fbfc140f0b93301646c0635dc (127.1.182.129:41945)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.130@KRBTEST.COM" created.
I20260928 02:04:19.059835 2330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:59813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
I20260928 02:04:19.080668 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:40465
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:19.404933 2525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:19.405205 2525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:19.405373 2525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:19.412815 2525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:19.412925 2525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:19.413121 2525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:04:19.422933 2525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:40465
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:19.424263 2525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:19.425592 2525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:19.439139 2530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:19.439042 2531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:19.439860 2533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:19.440313 2525 server_base.cc:1034] running on GCE node
I20260928 02:04:19.441130 2525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:19.442096 2525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:19.443327 2525 hybrid_clock.cc:648] HybridClock initialized: now 1790561059443280 us; error 39 us; skew 500 ppm
I20260928 02:04:19.443635 2525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:19 dist-test-slave-23bp krb5kdc[2285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561059, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:19.447005 2525 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:04:19.448714 2525 webserver.cc:460] Webserver started at http://127.1.182.130:39895/ using document root <none> and password file <none>
I20260928 02:04:19.449570 2525 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:19.449693 2525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:19.450011 2525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:19.453804 2525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "bd5fd970f7da47679785d421fc8b3966"
format_stamp: "Formatted at 2026-09-28 02:04:19 on dist-test-slave-23bp"
server_key: "7eb4eb400fe772513f12c37e99c3cb78"
server_key_iv: "72297366c2901578fb9582655b45ade7"
server_key_version: "encryptionkey@0"
I20260928 02:04:19.454540 2525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "bd5fd970f7da47679785d421fc8b3966"
format_stamp: "Formatted at 2026-09-28 02:04:19 on dist-test-slave-23bp"
server_key: "7eb4eb400fe772513f12c37e99c3cb78"
server_key_iv: "72297366c2901578fb9582655b45ade7"
server_key_version: "encryptionkey@0"
I20260928 02:04:19.459746 2525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:04:19.463270 2540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:19.464629 2525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:04:19.464838 2525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "bd5fd970f7da47679785d421fc8b3966"
format_stamp: "Formatted at 2026-09-28 02:04:19 on dist-test-slave-23bp"
server_key: "7eb4eb400fe772513f12c37e99c3cb78"
server_key_iv: "72297366c2901578fb9582655b45ade7"
server_key_version: "encryptionkey@0"
I20260928 02:04:19.465045 2525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:19.491796 2525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:19.495647 2525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:19.495934 2525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:19.496876 2525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:19.498255 2525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:19.498363 2525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:19.498498 2525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:19.498561 2525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:19.530658 2525 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:34653
I20260928 02:04:19.530733 2652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:34653 every 8 connection(s)
I20260928 02:04:19.531948 2525 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:04:19.538116 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 2525
I20260928 02:04:19.538321 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:04:19.538748 1754 external_mini_cluster.cc:1428] Setting key 549ec16a25cd587b1538e954b3e9e152
Sep 28 02:04:19 dist-test-slave-23bp krb5kdc[2285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561059, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:19.551553 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:19.534527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:34039 (local address 127.1.182.190:40465)
0928 02:04:19.534700 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:19.534704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:19.535947 (+ 1243us) server_negotiation.cc:407] Connection header received
0928 02:04:19.537735 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:19.537750 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:19.537837 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:19.537944 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:19.539649 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:19.540385 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:19.541892 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:19.542324 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:19.542451 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:19.545893 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:19.545898 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:19.545902 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:19.545932 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:19.548482 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:19.549208 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:19.549212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:19.549214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:19.549352 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:19.549787 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:19.549790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:19.549792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:19.550043 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:19.550189 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:19.550947 (+ 758us) server_negotiation.cc:299] Negotiation successful
0928 02:04:19.551204 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:04:19.552603 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:19.535027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:40465 (local address 127.1.182.130:34039)
0928 02:04:19.535771 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:19.535808 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:19.537470 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:19.538172 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:19.538180 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:19.538777 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:19.539484 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:19.539500 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:19.540537 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:19.540542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:19.541747 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:19.541758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:19.542551 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:19.542558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:19.542784 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:19.543520 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:19.543543 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:19.545653 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:19.548624 (+ 2971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:19.548634 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:19.548653 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:19.549077 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:19.549470 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:19.549476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:19.549480 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:19.549656 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:19.550163 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:19.550171 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:19.550582 (+ 411us) client_negotiation.cc:769] Sending connection context
0928 02:04:19.550862 (+ 280us) client_negotiation.cc:240] Negotiation successful
0928 02:04:19.551440 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":254,"threads_started":1}
I20260928 02:04:19.554070 2653 heartbeater.cc:347] Connected to a master server at 127.1.182.190:40465
I20260928 02:04:19.554382 2653 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.182.131@KRBTEST.COM; defaulting to no policy
I20260928 02:04:19.555133 2653 heartbeater.cc:511] Master 127.1.182.190:40465 requested a full tablet report, sending...
I20260928 02:04:19.556700 2330 ts_manager.cc:194] Registered new tserver with Master: bd5fd970f7da47679785d421fc8b3966 (127.1.182.130:34653)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.131@KRBTEST.COM" created.
I20260928 02:04:19.557613 2330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:34039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
I20260928 02:04:19.579123 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:40465
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:19.900979 2658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:19.901271 2658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:19.901367 2658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:19.908839 2658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:19.908964 2658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:19.909082 2658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:04:19.919209 2658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:40465
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:19.920462 2658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:19.921799 2658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:19.935204 2664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:19.935300 2663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:19.935279 2666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:19.936022 2658 server_base.cc:1034] running on GCE node
I20260928 02:04:19.936630 2658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:19.937594 2658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:19.938828 2658 hybrid_clock.cc:648] HybridClock initialized: now 1790561059938777 us; error 46 us; skew 500 ppm
I20260928 02:04:19.939218 2658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:19 dist-test-slave-23bp krb5kdc[2285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561059, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:19.942302 2658 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:04:19.943735 2658 webserver.cc:460] Webserver started at http://127.1.182.131:35463/ using document root <none> and password file <none>
I20260928 02:04:19.944391 2658 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:19.944483 2658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:19.944727 2658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:19.947656 2658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "5484809d15b949488bb13e24c43eb622"
format_stamp: "Formatted at 2026-09-28 02:04:19 on dist-test-slave-23bp"
server_key: "da190e7b664cc27fa54e58c7d8f00f2b"
server_key_iv: "9b8a8f659a565fa1e2da444802c5b7d5"
server_key_version: "encryptionkey@0"
I20260928 02:04:19.948385 2658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "5484809d15b949488bb13e24c43eb622"
format_stamp: "Formatted at 2026-09-28 02:04:19 on dist-test-slave-23bp"
server_key: "da190e7b664cc27fa54e58c7d8f00f2b"
server_key_iv: "9b8a8f659a565fa1e2da444802c5b7d5"
server_key_version: "encryptionkey@0"
I20260928 02:04:19.953411 2658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 02:04:19.957037 2673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:19.958321 2658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:04:19.958489 2658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "5484809d15b949488bb13e24c43eb622"
format_stamp: "Formatted at 2026-09-28 02:04:19 on dist-test-slave-23bp"
server_key: "da190e7b664cc27fa54e58c7d8f00f2b"
server_key_iv: "9b8a8f659a565fa1e2da444802c5b7d5"
server_key_version: "encryptionkey@0"
I20260928 02:04:19.958690 2658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:19.983764 2658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:19.987982 2658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:19.988276 2658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:19.989252 2658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:19.990633 2658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:19.990703 2658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:19.990836 2658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:19.990886 2658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:20.024108 2658 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:43763
I20260928 02:04:20.024196 2785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:43763 every 8 connection(s)
I20260928 02:04:20.025683 2658 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:04:20.025946 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 2658
I20260928 02:04:20.026137 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:04:20.026593 1754 external_mini_cluster.cc:1428] Setting key f03324514c66e8558f6472edf2da2501
Sep 28 02:04:20 dist-test-slave-23bp krb5kdc[2285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561059, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:20.047560 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:20.031319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:54815 (local address 127.1.182.190:40465)
0928 02:04:20.031523 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:20.031531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:20.031550 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:04:20.032932 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:20.032937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:20.033014 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:20.033125 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:20.034778 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.035633 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.037325 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.037734 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.037833 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:20.041430 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:20.041435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:20.041438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:20.041469 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:20.044082 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:20.045272 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:20.045279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:20.045282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:20.045415 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:20.045847 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:20.045852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:20.045855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:20.046123 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:20.046287 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:20.046962 (+ 675us) server_negotiation.cc:299] Negotiation successful
0928 02:04:20.047196 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:04:20.048599 2788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:20.029860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:40465 (local address 127.1.182.131:54815)
0928 02:04:20.030701 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:20.030753 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:20.032613 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:20.033356 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:20.033365 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:20.033971 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:20.034612 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:20.034631 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.035789 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.035794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:20.037186 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:20.037198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.037958 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.037963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:20.038151 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:20.038782 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:20.038804 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:20.041222 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:20.044234 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:20.044244 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:20.044262 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:20.045139 (+ 877us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:20.045530 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:20.045536 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:20.045540 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:20.045726 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:20.046271 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:20.046279 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:20.046621 (+ 342us) client_negotiation.cc:769] Sending connection context
0928 02:04:20.046894 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 02:04:20.047456 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":372,"threads_started":1}
I20260928 02:04:20.049911 2786 heartbeater.cc:347] Connected to a master server at 127.1.182.190:40465
I20260928 02:04:20.050211 2786 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:20.050917 2786 heartbeater.cc:511] Master 127.1.182.190:40465 requested a full tablet report, sending...
I20260928 02:04:20.052327 2330 ts_manager.cc:194] Registered new tserver with Master: 5484809d15b949488bb13e24c43eb622 (127.1.182.131:43763)
I20260928 02:04:20.053123 2330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:54815
I20260928 02:04:20.054818 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:04:20.062541 2519 heartbeater.cc:503] Master 127.1.182.190:40465 was elected leader, sending a full tablet report...
I20260928 02:04:20.074307 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:20.061049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34286 (local address 127.1.182.190:40465)
0928 02:04:20.061213 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:20.061217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:20.061783 (+ 566us) server_negotiation.cc:407] Connection header received
0928 02:04:20.062042 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:20.062046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:20.062104 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:20.062206 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:20.063840 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.064555 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.065812 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.066228 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.066363 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:20.069128 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:20.069134 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:20.069141 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:20.069176 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:20.071628 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:20.072346 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:20.072353 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:20.072356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:20.072483 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:20.072851 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:20.072857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:20.072860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:20.073184 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:20.073454 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:20.073782 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 02:04:20.073987 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:04:20.080931 2329 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:04:20.088249 2329 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:04:20.122542 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:20.100420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:43763 (local address 127.0.0.1:43160)
0928 02:04:20.101356 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:20.101395 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:20.101575 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:20.107203 (+ 5628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:20.107215 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:20.107271 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:20.107792 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:20.107809 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.112114 (+ 4305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.112124 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:20.116747 (+ 4623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:20.116769 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.121152 (+ 4383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.121162 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:20.121492 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:20.121579 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 02:04:20.121727 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 02:04:20.122000 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":266,"threads_started":1}
I20260928 02:04:20.125738 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:20.100705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.1.182.131:43763)
0928 02:04:20.106305 (+ 5600us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:20.106319 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:20.106364 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:04:20.106492 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:20.106505 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:20.106836 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:20.108055 (+ 1219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:20.109732 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.111915 (+ 2183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.117029 (+ 5114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.120867 (+ 3838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.123286 (+ 2419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:20.123622 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:20.124989 (+ 1367us) server_negotiation.cc:299] Negotiation successful
0928 02:04:20.125262 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5269,"thread_start_us":5113,"threads_started":1}
I20260928 02:04:20.130179 2721 tablet_service.cc:1467] Processing CreateTablet for tablet 19a5f11d94ef4dc585aa79e5fa4f2ab5 (DEFAULT_TABLE table=test-table [id=02adbcdd29bc421a8823b401952ed9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:20.131881 2721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19a5f11d94ef4dc585aa79e5fa4f2ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:20.133487 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:20.101897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58722 (local address 127.1.182.129:41945)
0928 02:04:20.126931 (+ 25034us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:20.126939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:20.126966 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:04:20.127118 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:20.127126 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:20.127365 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:20.127594 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:20.127975 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.129460 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.131220 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.132484 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.132640 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:20.132881 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:20.133024 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 02:04:20.133179 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24876,"thread_start_us":24710,"threads_started":1}
I20260928 02:04:20.133504 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:20.101179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:41945 (local address 127.0.0.1:58722)
0928 02:04:20.122898 (+ 21719us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:20.122923 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:20.123054 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:20.127586 (+ 4532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:20.127591 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:20.127616 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:20.127821 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:20.127832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.129602 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.129611 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:20.131009 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:20.131024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.132660 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.132666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:20.132838 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:20.132876 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:04:20.132976 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 02:04:20.133144 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21570,"mutex_wait_us":64,"thread_start_us":25138,"threads_started":1}
I20260928 02:04:20.134907 2801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:20.101257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:34653 (local address 127.0.0.1:37406)
0928 02:04:20.126050 (+ 24793us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:20.126078 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:20.126255 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:20.127112 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:20.127117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:20.127141 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:20.127481 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:20.127495 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.129413 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.129420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:20.131603 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:20.131618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.134088 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.134095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:20.134283 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:20.134328 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 02:04:20.134419 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 02:04:20.134548 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24647,"mutex_wait_us":79,"thread_start_us":24489,"threads_started":1}
I20260928 02:04:20.135218 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:20.101583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37406 (local address 127.1.182.130:34653)
0928 02:04:20.125357 (+ 23774us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:20.125370 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:20.126430 (+ 1060us) server_negotiation.cc:407] Connection header received
0928 02:04:20.126568 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:20.126581 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:20.126869 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:20.127112 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:20.127692 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.129279 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.131787 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.133605 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.133742 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:20.133917 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:20.134508 (+ 591us) server_negotiation.cc:299] Negotiation successful
0928 02:04:20.134681 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23534,"thread_start_us":23809,"threads_started":1}
I20260928 02:04:20.137355 2454 tablet_service.cc:1467] Processing CreateTablet for tablet 19a5f11d94ef4dc585aa79e5fa4f2ab5 (DEFAULT_TABLE table=test-table [id=02adbcdd29bc421a8823b401952ed9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:20.138571 2454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19a5f11d94ef4dc585aa79e5fa4f2ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:20.138886 2588 tablet_service.cc:1467] Processing CreateTablet for tablet 19a5f11d94ef4dc585aa79e5fa4f2ab5 (DEFAULT_TABLE table=test-table [id=02adbcdd29bc421a8823b401952ed9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:20.140228 2588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19a5f11d94ef4dc585aa79e5fa4f2ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:20.147385 2804 tablet_bootstrap.cc:492] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622: Bootstrap starting.
I20260928 02:04:20.151175 2805 tablet_bootstrap.cc:492] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 527b257fbfc140f0b93301646c0635dc: Bootstrap starting.
I20260928 02:04:20.152496 2804 tablet_bootstrap.cc:654] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:20.153494 2804 log.cc:824] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:20.154762 2805 tablet_bootstrap.cc:654] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 527b257fbfc140f0b93301646c0635dc: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:20.156103 2805 log.cc:824] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 527b257fbfc140f0b93301646c0635dc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:20.155941 2806 tablet_bootstrap.cc:492] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P bd5fd970f7da47679785d421fc8b3966: Bootstrap starting.
I20260928 02:04:20.158638 2804 tablet_bootstrap.cc:492] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622: No bootstrap required, opened a new log
I20260928 02:04:20.158928 2804 ts_tablet_manager.cc:1397] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260928 02:04:20.159425 2805 tablet_bootstrap.cc:492] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 527b257fbfc140f0b93301646c0635dc: No bootstrap required, opened a new log
I20260928 02:04:20.159684 2805 ts_tablet_manager.cc:1397] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 527b257fbfc140f0b93301646c0635dc: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260928 02:04:20.160444 2806 tablet_bootstrap.cc:654] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P bd5fd970f7da47679785d421fc8b3966: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:20.161615 2806 log.cc:824] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P bd5fd970f7da47679785d421fc8b3966: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:20.166975 2804 raft_consensus.cc:348] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5484809d15b949488bb13e24c43eb622" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 43763 } } peers { permanent_uuid: "bd5fd970f7da47679785d421fc8b3966" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 34653 } } peers { permanent_uuid: "527b257fbfc140f0b93301646c0635dc" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 41945 } }
I20260928 02:04:20.167948 2804 raft_consensus.cc:374] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:20.168062 2804 raft_consensus.cc:729] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5484809d15b949488bb13e24c43eb622, State: Initialized, Role: FOLLOWER
I20260928 02:04:20.168751 2804 consensus_queue.cc:260] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5484809d15b949488bb13e24c43eb622" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 43763 } } peers { permanent_uuid: "bd5fd970f7da47679785d421fc8b3966" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 34653 } } peers { permanent_uuid: "527b257fbfc140f0b93301646c0635dc" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 41945 } }
I20260928 02:04:20.169878 2786 heartbeater.cc:503] Master 127.1.182.190:40465 was elected leader, sending a full tablet report...
I20260928 02:04:20.170538 2806 tablet_bootstrap.cc:492] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P bd5fd970f7da47679785d421fc8b3966: No bootstrap required, opened a new log
I20260928 02:04:20.170524 2804 ts_tablet_manager.cc:1428] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260928 02:04:20.170794 2806 ts_tablet_manager.cc:1397] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P bd5fd970f7da47679785d421fc8b3966: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260928 02:04:20.172197 2805 raft_consensus.cc:348] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 527b257fbfc140f0b93301646c0635dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5484809d15b949488bb13e24c43eb622" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 43763 } } peers { permanent_uuid: "bd5fd970f7da47679785d421fc8b3966" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 34653 } } peers { permanent_uuid: "527b257fbfc140f0b93301646c0635dc" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 41945 } }
I20260928 02:04:20.172649 2805 raft_consensus.cc:374] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 527b257fbfc140f0b93301646c0635dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:20.172748 2805 raft_consensus.cc:729] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 527b257fbfc140f0b93301646c0635dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 527b257fbfc140f0b93301646c0635dc, State: Initialized, Role: FOLLOWER
I20260928 02:04:20.173429 2805 consensus_queue.cc:260] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 527b257fbfc140f0b93301646c0635dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5484809d15b949488bb13e24c43eb622" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 43763 } } peers { permanent_uuid: "bd5fd970f7da47679785d421fc8b3966" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 34653 } } peers { permanent_uuid: "527b257fbfc140f0b93301646c0635dc" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 41945 } }
I20260928 02:04:20.175464 2805 ts_tablet_manager.cc:1428] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 527b257fbfc140f0b93301646c0635dc: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260928 02:04:20.180296 2806 raft_consensus.cc:348] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P bd5fd970f7da47679785d421fc8b3966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5484809d15b949488bb13e24c43eb622" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 43763 } } peers { permanent_uuid: "bd5fd970f7da47679785d421fc8b3966" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 34653 } } peers { permanent_uuid: "527b257fbfc140f0b93301646c0635dc" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 41945 } }
I20260928 02:04:20.180753 2806 raft_consensus.cc:374] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P bd5fd970f7da47679785d421fc8b3966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:20.180850 2806 raft_consensus.cc:729] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P bd5fd970f7da47679785d421fc8b3966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd5fd970f7da47679785d421fc8b3966, State: Initialized, Role: FOLLOWER
I20260928 02:04:20.181481 2806 consensus_queue.cc:260] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P bd5fd970f7da47679785d421fc8b3966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5484809d15b949488bb13e24c43eb622" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 43763 } } peers { permanent_uuid: "bd5fd970f7da47679785d421fc8b3966" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 34653 } } peers { permanent_uuid: "527b257fbfc140f0b93301646c0635dc" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 41945 } }
I20260928 02:04:20.182438 2653 heartbeater.cc:503] Master 127.1.182.190:40465 was elected leader, sending a full tablet report...
I20260928 02:04:20.182929 2806 ts_tablet_manager.cc:1428] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P bd5fd970f7da47679785d421fc8b3966: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260928 02:04:20.205039 2810 raft_consensus.cc:482] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:20.205404 2810 raft_consensus.cc:504] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5484809d15b949488bb13e24c43eb622" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 43763 } } peers { permanent_uuid: "bd5fd970f7da47679785d421fc8b3966" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 34653 } } peers { permanent_uuid: "527b257fbfc140f0b93301646c0635dc" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 41945 } }
I20260928 02:04:20.207307 2810 leader_election.cc:288] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd5fd970f7da47679785d421fc8b3966 (127.1.182.130:34653), 527b257fbfc140f0b93301646c0635dc (127.1.182.129:41945)
I20260928 02:04:20.212806 2788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:20.207494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:34653 (local address 127.1.182.131:38293)
0928 02:04:20.207869 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:20.207891 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:20.208053 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:20.208399 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:20.208404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:20.208436 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:20.208612 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:20.208620 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.209593 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.209598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:20.210863 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:20.210874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.212009 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.212014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:20.212264 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:20.212288 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:04:20.212392 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:04:20.212521 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226}
I20260928 02:04:20.212806 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:20.207619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:38293 (local address 127.1.182.130:34653)
0928 02:04:20.207823 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:20.207830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:20.207980 (+ 150us) server_negotiation.cc:407] Connection header received
0928 02:04:20.208194 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:20.208199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:20.208268 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:20.208367 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:20.208744 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.209448 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.211024 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.211857 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.212013 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:20.212087 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:20.212421 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 02:04:20.212542 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:04:20.213909 2608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19a5f11d94ef4dc585aa79e5fa4f2ab5" candidate_uuid: "5484809d15b949488bb13e24c43eb622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd5fd970f7da47679785d421fc8b3966" is_pre_election: true
I20260928 02:04:20.214372 2813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:20.207729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:41945 (local address 127.1.182.131:41625)
0928 02:04:20.208582 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:20.208596 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:20.208732 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:20.209045 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:20.209050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:20.209070 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:20.209406 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:20.209413 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.210417 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.210423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:20.212253 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:20.212264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.213721 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.213725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:20.213841 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:20.213863 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:04:20.213943 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:04:20.214025 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":51,"thread_start_us":446,"threads_started":1}
I20260928 02:04:20.214375 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:20.207736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:41625 (local address 127.1.182.129:41945)
0928 02:04:20.207948 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:20.207956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:20.208687 (+ 731us) server_negotiation.cc:407] Connection header received
0928 02:04:20.208878 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:20.208882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:20.208939 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:20.209034 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:20.209558 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.210275 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.212406 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.213551 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.213727 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:20.213804 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:20.213996 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 02:04:20.214137 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:04:20.214496 2608 raft_consensus.cc:2454] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P bd5fd970f7da47679785d421fc8b3966 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5484809d15b949488bb13e24c43eb622 in term 0.
I20260928 02:04:20.215301 2675 leader_election.cc:302] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5484809d15b949488bb13e24c43eb622, bd5fd970f7da47679785d421fc8b3966; no voters:
I20260928 02:04:20.215543 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19a5f11d94ef4dc585aa79e5fa4f2ab5" candidate_uuid: "5484809d15b949488bb13e24c43eb622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "527b257fbfc140f0b93301646c0635dc" is_pre_election: true
I20260928 02:04:20.215734 2810 raft_consensus.cc:2781] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:04:20.215902 2810 raft_consensus.cc:482] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:04:20.216006 2810 raft_consensus.cc:3037] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:20.216109 2474 raft_consensus.cc:2454] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 527b257fbfc140f0b93301646c0635dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5484809d15b949488bb13e24c43eb622 in term 0.
I20260928 02:04:20.218315 2810 raft_consensus.cc:504] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5484809d15b949488bb13e24c43eb622" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 43763 } } peers { permanent_uuid: "bd5fd970f7da47679785d421fc8b3966" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 34653 } } peers { permanent_uuid: "527b257fbfc140f0b93301646c0635dc" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 41945 } }
I20260928 02:04:20.218885 2810 leader_election.cc:288] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622 [CANDIDATE]: Term 1 election: Requested vote from peers bd5fd970f7da47679785d421fc8b3966 (127.1.182.130:34653), 527b257fbfc140f0b93301646c0635dc (127.1.182.129:41945)
I20260928 02:04:20.219311 2608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19a5f11d94ef4dc585aa79e5fa4f2ab5" candidate_uuid: "5484809d15b949488bb13e24c43eb622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd5fd970f7da47679785d421fc8b3966"
I20260928 02:04:20.219395 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19a5f11d94ef4dc585aa79e5fa4f2ab5" candidate_uuid: "5484809d15b949488bb13e24c43eb622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "527b257fbfc140f0b93301646c0635dc"
I20260928 02:04:20.219509 2608 raft_consensus.cc:3037] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P bd5fd970f7da47679785d421fc8b3966 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:20.219594 2474 raft_consensus.cc:3037] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 527b257fbfc140f0b93301646c0635dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:20.221670 2608 raft_consensus.cc:2454] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P bd5fd970f7da47679785d421fc8b3966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5484809d15b949488bb13e24c43eb622 in term 1.
I20260928 02:04:20.221717 2474 raft_consensus.cc:2454] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 527b257fbfc140f0b93301646c0635dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5484809d15b949488bb13e24c43eb622 in term 1.
I20260928 02:04:20.222055 2675 leader_election.cc:302] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5484809d15b949488bb13e24c43eb622, bd5fd970f7da47679785d421fc8b3966; no voters:
I20260928 02:04:20.222359 2810 raft_consensus.cc:2781] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:20.222765 2810 raft_consensus.cc:686] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622 [term 1 LEADER]: Becoming Leader. State: Replica: 5484809d15b949488bb13e24c43eb622, State: Running, Role: LEADER
I20260928 02:04:20.223287 2810 consensus_queue.cc:237] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5484809d15b949488bb13e24c43eb622" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 43763 } } peers { permanent_uuid: "bd5fd970f7da47679785d421fc8b3966" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 34653 } } peers { permanent_uuid: "527b257fbfc140f0b93301646c0635dc" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 41945 } }
I20260928 02:04:20.227779 2330 catalog_manager.cc:5495] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5484809d15b949488bb13e24c43eb622 (127.1.182.131). New cstate: current_term: 1 leader_uuid: "5484809d15b949488bb13e24c43eb622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5484809d15b949488bb13e24c43eb622" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 43763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd5fd970f7da47679785d421fc8b3966" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 34653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "527b257fbfc140f0b93301646c0635dc" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 41945 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:04:20 dist-test-slave-23bp krb5kdc[2285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:04:20.276687 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:20.257179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58728 (local address 127.1.182.129:41945)
0928 02:04:20.257412 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:20.257418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:20.257843 (+ 425us) server_negotiation.cc:407] Connection header received
0928 02:04:20.258049 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:20.258061 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:20.258131 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:20.258235 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:20.259708 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.260439 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.261680 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.262126 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.262264 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:20.268196 (+ 5932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:20.268218 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:20.268246 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:20.268286 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:20.273211 (+ 4925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:20.274236 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:20.274246 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:20.274249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:20.274634 (+ 385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:20.275161 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:20.275165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:20.275167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:20.275521 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:20.275973 (+ 452us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:20.276215 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 02:04:20.276371 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 02:04:20.277683 2787 tablet.cc:2369] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:04:20.282969 2520 tablet.cc:2369] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 527b257fbfc140f0b93301646c0635dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:04:20.283939 2654 tablet.cc:2369] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P bd5fd970f7da47679785d421fc8b3966: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:04:20 dist-test-slave-23bp krb5kdc[2285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561060, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:04:20 dist-test-slave-23bp krb5kdc[2285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561060, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:04:20.330291 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:20.316354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58734 (local address 127.1.182.129:41945)
0928 02:04:20.316570 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:20.316579 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:20.316989 (+ 410us) server_negotiation.cc:407] Connection header received
0928 02:04:20.317222 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:20.317228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:20.317302 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:20.317407 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:20.318732 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.319566 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.320912 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.321369 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.321511 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:20.325313 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:20.325318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:20.325322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:20.325356 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:20.327760 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:20.328384 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:20.328391 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:20.328394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:20.328521 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:20.328889 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:20.328896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:20.328900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:20.329167 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:20.329492 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:20.329725 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 02:04:20.329938 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 02:04:20 dist-test-slave-23bp krb5kdc[2285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:04:20 dist-test-slave-23bp krb5kdc[2285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:04:20.374291 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:20.360338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58750 (local address 127.1.182.129:41945)
0928 02:04:20.360570 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:20.360577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:20.360733 (+ 156us) server_negotiation.cc:407] Connection header received
0928 02:04:20.360958 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:20.360963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:20.361023 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:20.361126 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:20.362453 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.363264 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.364455 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.364868 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.364997 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:20.368905 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:20.368910 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:20.368913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:20.368951 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:20.371807 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:20.372430 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:20.372437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:20.372441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:20.372556 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:20.372901 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:20.372908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:20.372911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:20.373197 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:20.373439 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:20.373713 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 02:04:20.373943 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260928 02:04:20.376075 2434 tablet_service.cc:2908] Rejecting scan request for tablet 19a5f11d94ef4dc585aa79e5fa4f2ab5: Uninitialized: clean time has not yet been initialized
I20260928 02:04:20.378970 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 2391
I20260928 02:04:20.390874 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:20.379816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58752 (local address 127.1.182.129:41945)
0928 02:04:20.380013 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:20.380022 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:20.380293 (+ 271us) server_negotiation.cc:407] Connection header received
0928 02:04:20.380426 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:20.380430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:20.380495 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:20.380572 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:20.381726 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.382437 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.383626 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.384045 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.384145 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:20.386199 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:20.386203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:20.386206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:20.386229 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:20.388718 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:20.389298 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:20.389302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:20.389304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:20.389413 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:20.389693 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:20.389696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:20.389698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:20.389956 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:20.390146 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:20.390377 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 02:04:20.390525 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:04:20.391554 2514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:20.483944 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 2391
I20260928 02:04:20.500836 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 2525
Sep 28 02:04:20 dist-test-slave-23bp krb5kdc[2285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.130@KRBTEST.COM
I20260928 02:04:20.515452 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:20.501588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37422 (local address 127.1.182.130:34653)
0928 02:04:20.501792 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:20.501798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:20.501862 (+ 64us) server_negotiation.cc:407] Connection header received
0928 02:04:20.502072 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:20.502076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:20.502136 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:20.502258 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:20.503479 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.504260 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.505496 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.505899 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.506033 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:20.510263 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:20.510279 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:20.510294 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:20.510329 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:20.513025 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:20.513726 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:20.513734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:20.513736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:20.513859 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:20.514163 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:20.514166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:20.514168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:20.514526 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:20.514755 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:20.514972 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 02:04:20.515199 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 02:04:20.516100 2648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:20.605561 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 2525
W20260928 02:04:20.622416 2675 connection.cc:535] client connection to 127.1.182.130:34653 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:04:20.622819 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 2658
W20260928 02:04:20.622745 2675 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:04:20.624321 2813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:20.623940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:34653 (local address 127.1.182.131:42757)
0928 02:04:20.624090 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:20.624189 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.130:34653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 02:04:20.624558 2675 consensus_peers.cc:487] T 19a5f11d94ef4dc585aa79e5fa4f2ab5 P 5484809d15b949488bb13e24c43eb622 -> Peer bd5fd970f7da47679785d421fc8b3966 (127.1.182.130:34653): Couldn't send request to peer bd5fd970f7da47679785d421fc8b3966. Status: Network error: Client connection negotiation failed: client connection to 127.1.182.130:34653: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Sep 28 02:04:20 dist-test-slave-23bp krb5kdc[2285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.131@KRBTEST.COM
I20260928 02:04:20.637630 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:20.623619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43166 (local address 127.1.182.131:43763)
0928 02:04:20.623803 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:20.623810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:20.623834 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:04:20.624046 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:20.624052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:20.624123 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:20.624233 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:20.625546 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.626262 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.627679 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:20.628083 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:20.628220 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:20.632229 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:20.632240 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:20.632256 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:20.632287 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:20.635038 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:20.635816 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:20.635823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:20.635826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:20.635968 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:20.636296 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:20.636299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:20.636301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:20.636678 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:20.636903 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:20.637104 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 02:04:20.637291 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:04:20.638289 2781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:20.726518 2813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:20.726087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:41945 (local address 127.1.182.131:45079)
0928 02:04:20.726251 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:20.726351 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 02:04:20.726565 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 2658
I20260928 02:04:20.743227 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 2298
I20260928 02:04:20.744350 2360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:20.850322 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 2298
2026-09-28T02:04:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4136 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:04:20 dist-test-slave-23bp krb5kdc[2834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:04:20 dist-test-slave-23bp krb5kdc[2834](info): set up 2 sockets
Sep 28 02:04:20 dist-test-slave-23bp krb5kdc[2834](info): commencing operation
krb5kdc: starting...
W20260928 02:04:21.985886 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:04:22 dist-test-slave-23bp krb5kdc[2834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:04:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:04:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:04:22.166734 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:39221
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:39221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:22.503772 2847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:22.504099 2847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:22.504254 2847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:22.511579 2847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:22.511741 2847 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:04:22.511829 2847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:22.511921 2847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:22.512015 2847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:22.512106 2847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:22.522132 2847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:39221
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:39221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:22.523500 2847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:22.524852 2847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:22.536813 2852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:22.536760 2853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:22.536840 2855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:22.537808 2847 server_base.cc:1034] running on GCE node
I20260928 02:04:22.538566 2847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:22.540043 2847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:22.541287 2847 hybrid_clock.cc:648] HybridClock initialized: now 1790561062541214 us; error 56 us; skew 500 ppm
I20260928 02:04:22.541605 2847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:22 dist-test-slave-23bp krb5kdc[2834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561062, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:22.544744 2847 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:04:22.546131 2847 webserver.cc:460] Webserver started at http://127.1.182.190:44485/ using document root <none> and password file <none>
I20260928 02:04:22.546782 2847 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:22.546866 2847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:22.547210 2847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:22.550099 2847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "2b869a603f074509bb36b8ffb3102229"
format_stamp: "Formatted at 2026-09-28 02:04:22 on dist-test-slave-23bp"
server_key: "99bcff6e3beb373c65c7eccf6c0065a0"
server_key_iv: "70bbf68744f1dd1f4774f15cac22342b"
server_key_version: "encryptionkey@0"
I20260928 02:04:22.550853 2847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "2b869a603f074509bb36b8ffb3102229"
format_stamp: "Formatted at 2026-09-28 02:04:22 on dist-test-slave-23bp"
server_key: "99bcff6e3beb373c65c7eccf6c0065a0"
server_key_iv: "70bbf68744f1dd1f4774f15cac22342b"
server_key_version: "encryptionkey@0"
I20260928 02:04:22.556180 2847 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 02:04:22.559547 2862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:22.560885 2847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:04:22.561026 2847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "2b869a603f074509bb36b8ffb3102229"
format_stamp: "Formatted at 2026-09-28 02:04:22 on dist-test-slave-23bp"
server_key: "99bcff6e3beb373c65c7eccf6c0065a0"
server_key_iv: "70bbf68744f1dd1f4774f15cac22342b"
server_key_version: "encryptionkey@0"
I20260928 02:04:22.561203 2847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:22.581162 2847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:22.585237 2847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:22.585455 2847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:22.606621 2847 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:39221
I20260928 02:04:22.606643 2923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:39221 every 8 connection(s)
I20260928 02:04:22.608013 2847 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:04:22.611850 2924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:22.613670 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 2847
I20260928 02:04:22.613905 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:04:22.614538 1754 external_mini_cluster.cc:1428] Setting key b396d54411c11d164fedc6e5462a4f8a
I20260928 02:04:22.624643 2924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229: Bootstrap starting.
I20260928 02:04:22.628222 2924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:22.629227 2924 log.cc:824] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:04:22 dist-test-slave-23bp krb5kdc[2834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:22.632630 2924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229: No bootstrap required, opened a new log
I20260928 02:04:22.639122 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:22.616481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.1.182.190:39221)
0928 02:04:22.617307 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:22.617322 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:22.617381 (+ 59us) server_negotiation.cc:407] Connection header received
0928 02:04:22.619783 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:22.619810 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:22.620314 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:22.620755 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:22.621981 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:22.623958 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:22.625182 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:22.626204 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:22.626431 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:22.630633 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:22.630644 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:22.630663 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:22.630700 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:22.634515 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:22.635260 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:22.635269 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:22.635277 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:22.635485 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:22.635851 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:22.635856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:22.635860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:22.636579 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:22.636869 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:22.637226 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 02:04:22.637659 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":331,"threads_started":1}
I20260928 02:04:22.642894 2924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b869a603f074509bb36b8ffb3102229" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39221 } }
I20260928 02:04:22.643287 2924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:22.643366 2924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b869a603f074509bb36b8ffb3102229, State: Initialized, Role: FOLLOWER
I20260928 02:04:22.643904 2924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b869a603f074509bb36b8ffb3102229" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39221 } }
I20260928 02:04:22.644104 2924 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:04:22.644200 2924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:04:22.644352 2924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:22.646734 2924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b869a603f074509bb36b8ffb3102229" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39221 } }
I20260928 02:04:22.647228 2924 leader_election.cc:302] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b869a603f074509bb36b8ffb3102229; no voters:
I20260928 02:04:22.647804 2924 leader_election.cc:288] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:04:22.647872 2929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:22.648619 2929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229 [term 1 LEADER]: Becoming Leader. State: Replica: 2b869a603f074509bb36b8ffb3102229, State: Running, Role: LEADER
I20260928 02:04:22.649168 2929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b869a603f074509bb36b8ffb3102229" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39221 } }
I20260928 02:04:22.649968 2924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:22.651927 2930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b869a603f074509bb36b8ffb3102229" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b869a603f074509bb36b8ffb3102229" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39221 } } }
I20260928 02:04:22.652464 2930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:22.653290 2937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:22.653139 2931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b869a603f074509bb36b8ffb3102229. Latest consensus state: current_term: 1 leader_uuid: "2b869a603f074509bb36b8ffb3102229" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b869a603f074509bb36b8ffb3102229" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39221 } } }
I20260928 02:04:22.653506 2931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:22.659546 2937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:22.669325 2937 catalog_manager.cc:1347] Generated new cluster ID: 0ae76c18b31845119b7f3629bbce6a1e
I20260928 02:04:22.669448 2937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:22.712349 2937 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:04:22.713706 2937 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:22.724370 2937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229: Generated new TSK 0
I20260928 02:04:22.725230 2937 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.182.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
I20260928 02:04:22.778949 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39221
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:23.111354 2950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:23.111646 2950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:23.111742 2950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:23.119786 2950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:23.119917 2950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:23.119997 2950 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:04:23.120085 2950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:04:23.129925 2950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.182.190:39221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:23.131219 2950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:23.132512 2950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:23.145736 2956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:23.145671 2955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:23.146456 2958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:23.146656 2950 server_base.cc:1034] running on GCE node
I20260928 02:04:23.147416 2950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:23.148372 2950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:23.149609 2950 hybrid_clock.cc:648] HybridClock initialized: now 1790561063149554 us; error 47 us; skew 500 ppm
I20260928 02:04:23.149983 2950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:23 dist-test-slave-23bp krb5kdc[2834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561063, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:23.153016 2950 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:04:23.154385 2950 webserver.cc:460] Webserver started at http://127.1.182.129:34423/ using document root <none> and password file <none>
I20260928 02:04:23.155138 2950 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:23.155264 2950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:23.155592 2950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:23.158387 2950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "95d757d0b10c450b8b6690168898b589"
format_stamp: "Formatted at 2026-09-28 02:04:23 on dist-test-slave-23bp"
server_key: "04c6a02d8b3db987b8350a359201ee10"
server_key_iv: "0df0677ba7be27880a1da95e9d335b59"
server_key_version: "encryptionkey@0"
I20260928 02:04:23.159201 2950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "95d757d0b10c450b8b6690168898b589"
format_stamp: "Formatted at 2026-09-28 02:04:23 on dist-test-slave-23bp"
server_key: "04c6a02d8b3db987b8350a359201ee10"
server_key_iv: "0df0677ba7be27880a1da95e9d335b59"
server_key_version: "encryptionkey@0"
I20260928 02:04:23.164315 2950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 02:04:23.168035 2965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:23.169324 2950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:04:23.169488 2950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "95d757d0b10c450b8b6690168898b589"
format_stamp: "Formatted at 2026-09-28 02:04:23 on dist-test-slave-23bp"
server_key: "04c6a02d8b3db987b8350a359201ee10"
server_key_iv: "0df0677ba7be27880a1da95e9d335b59"
server_key_version: "encryptionkey@0"
I20260928 02:04:23.169698 2950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:23.189983 2950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:23.194087 2950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:23.194375 2950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:23.196966 2950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:23.197077 2950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:23.197235 2950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:23.197312 2950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:04:23 dist-test-slave-23bp krb5kdc[2834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561063, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:23.222189 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:23.197790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:36291 (local address 127.1.182.190:39221)
0928 02:04:23.198311 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:23.198320 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:23.199443 (+ 1123us) server_negotiation.cc:407] Connection header received
0928 02:04:23.201605 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:23.201615 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:23.201727 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:23.201848 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:23.204185 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:23.205315 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:23.207874 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:23.208449 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:23.208585 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:23.214563 (+ 5978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:23.214569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:23.214574 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:23.214616 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:23.217702 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:23.219165 (+ 1463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:23.219170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:23.219174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:23.219331 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:23.220067 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:23.220074 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:23.220077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:23.220493 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:23.220673 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:23.221590 (+ 917us) server_negotiation.cc:299] Negotiation successful
0928 02:04:23.221807 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":279,"threads_started":1}
I20260928 02:04:23.223706 2975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:23.198370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39221 (local address 127.1.182.129:36291)
0928 02:04:23.199265 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:23.199316 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:23.201303 (+ 1987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:23.202120 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:23.202132 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:23.202889 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:23.203973 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:23.204008 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:23.205470 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:23.205475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:23.207647 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:23.207658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:23.208772 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:23.208782 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:23.209184 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:23.210164 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:23.210214 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:23.214253 (+ 4039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:23.217859 (+ 3606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:23.217871 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:23.217906 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:23.218762 (+ 856us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:23.219476 (+ 714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:23.219484 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:23.219493 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:23.219899 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:23.220661 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:23.220668 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:23.221207 (+ 539us) client_negotiation.cc:769] Sending connection context
0928 02:04:23.221499 (+ 292us) client_negotiation.cc:240] Negotiation successful
0928 02:04:23.222114 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":363,"threads_started":1}
I20260928 02:04:23.245725 2950 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:40771
I20260928 02:04:23.245743 3085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:40771 every 8 connection(s)
I20260928 02:04:23.247392 2950 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:04:23.248943 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:23.233333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:35249 (local address 127.1.182.190:39221)
0928 02:04:23.233551 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:23.233559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:23.234530 (+ 971us) server_negotiation.cc:407] Connection header received
0928 02:04:23.234742 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:23.234750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:23.234855 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:23.234963 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:23.236292 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:23.237553 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:23.238849 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:23.239651 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:23.239792 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:23.242745 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:23.242755 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:23.242763 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:23.242814 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:23.246372 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:23.247002 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:23.247007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:23.247010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:23.247183 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:23.247523 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:23.247529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:23.247533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:23.247791 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:23.247934 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:23.248380 (+ 446us) server_negotiation.cc:299] Negotiation successful
0928 02:04:23.248604 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 02:04:23.249097 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:23.233426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39221 (local address 127.1.182.129:35249)
0928 02:04:23.234389 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:23.234412 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:23.234576 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:23.235017 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:23.235027 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:23.235800 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:23.236135 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:23.236145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:23.237715 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:23.237719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:23.238709 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:23.238721 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:23.239908 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:23.239915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:23.240169 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:23.241072 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:23.241103 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:23.242490 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:23.246495 (+ 4005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:23.246500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:23.246502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:23.246879 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:23.247289 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:23.247292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:23.247295 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:23.247409 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:23.247919 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:23.247924 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:23.248115 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 02:04:23.248269 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 02:04:23.248619 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":675,"threads_started":1}
I20260928 02:04:23.249992 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 2950
I20260928 02:04:23.250229 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:04:23.249996 3086 heartbeater.cc:347] Connected to a master server at 127.1.182.190:39221
I20260928 02:04:23.250985 1754 external_mini_cluster.cc:1428] Setting key 2eec8a07a11793ad921f201fb82bc43a
I20260928 02:04:23.251570 3086 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:23.252437 3086 heartbeater.cc:511] Master 127.1.182.190:39221 requested a full tablet report, sending...
I20260928 02:04:23.254549 2879 ts_manager.cc:194] Registered new tserver with Master: 95d757d0b10c450b8b6690168898b589 (127.1.182.129:40771)
I20260928 02:04:23.256035 2879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:36291
WARNING: no policy specified for kudu/127.1.182.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
I20260928 02:04:23.293854 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39221
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:23.611400 3090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:23.611709 3090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:23.611883 3090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:23.619197 3090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:23.619331 3090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:23.619480 3090 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:04:23.619622 3090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:04:23.629402 3090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.182.190:39221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:23.630733 3090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:23.632128 3090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:23.645203 3095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:23.645418 3096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:23.645305 3098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:23.646241 3090 server_base.cc:1034] running on GCE node
I20260928 02:04:23.646862 3090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:23.647845 3090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:23.649060 3090 hybrid_clock.cc:648] HybridClock initialized: now 1790561063649000 us; error 29 us; skew 500 ppm
I20260928 02:04:23.649310 3090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:23 dist-test-slave-23bp krb5kdc[2834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561063, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:23.658069 3090 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:04:23.659614 3090 webserver.cc:460] Webserver started at http://127.1.182.130:41535/ using document root <none> and password file <none>
I20260928 02:04:23.660358 3090 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:23.660467 3090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:23.660794 3090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:23.663686 3090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "9f27378c91b242c4913a2035f885ce34"
format_stamp: "Formatted at 2026-09-28 02:04:23 on dist-test-slave-23bp"
server_key: "557a63571eacf4003b97e546b9a22e01"
server_key_iv: "be155c8a6498f4f818c44bac0a099345"
server_key_version: "encryptionkey@0"
I20260928 02:04:23.664543 3090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "9f27378c91b242c4913a2035f885ce34"
format_stamp: "Formatted at 2026-09-28 02:04:23 on dist-test-slave-23bp"
server_key: "557a63571eacf4003b97e546b9a22e01"
server_key_iv: "be155c8a6498f4f818c44bac0a099345"
server_key_version: "encryptionkey@0"
I20260928 02:04:23.669847 3090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 02:04:23.673491 3105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:23.675040 3090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:04:23.675283 3090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "9f27378c91b242c4913a2035f885ce34"
format_stamp: "Formatted at 2026-09-28 02:04:23 on dist-test-slave-23bp"
server_key: "557a63571eacf4003b97e546b9a22e01"
server_key_iv: "be155c8a6498f4f818c44bac0a099345"
server_key_version: "encryptionkey@0"
I20260928 02:04:23.675508 3090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:23.704311 3090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:23.708228 3090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:23.708504 3090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:23.711145 3090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:23.711256 3090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:23.711426 3090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:23.711565 3090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:04:23 dist-test-slave-23bp krb5kdc[2834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561063, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:23.736013 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:23.711770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:47489 (local address 127.1.182.190:39221)
0928 02:04:23.711941 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:23.711947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:23.713483 (+ 1536us) server_negotiation.cc:407] Connection header received
0928 02:04:23.716576 (+ 3093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:23.716585 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:23.716689 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:23.716801 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:23.719683 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:23.720959 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:23.723242 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:23.723853 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:23.723989 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:23.728617 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:23.728627 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:23.728635 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:23.728688 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:23.732579 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:23.733265 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:23.733273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:23.733291 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:23.733450 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:23.733877 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:23.733885 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:23.733891 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:23.734328 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:23.734502 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:23.735198 (+ 696us) server_negotiation.cc:299] Negotiation successful
0928 02:04:23.735459 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:04:23.736733 3114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:23.712477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39221 (local address 127.1.182.130:47489)
0928 02:04:23.713287 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:23.713345 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:23.716153 (+ 2808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:23.717407 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:23.717423 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:23.718498 (+ 1075us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:23.719430 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:23.719477 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:23.721125 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:23.721131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:23.723015 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:23.723030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:23.724098 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:23.724105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:23.724334 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:23.725170 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:23.725205 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:23.728336 (+ 3131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:23.732728 (+ 4392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:23.732735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:23.732747 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:23.733154 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:23.733563 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:23.733566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:23.733568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:23.733760 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:23.734454 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:23.734459 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:23.734783 (+ 324us) client_negotiation.cc:769] Sending connection context
0928 02:04:23.734998 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 02:04:23.735570 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":322,"threads_started":1}
I20260928 02:04:23.756943 3172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:23.743286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39221 (local address 127.1.182.130:49119)
0928 02:04:23.743908 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:23.743928 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:23.744080 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:23.744486 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:23.744492 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:23.745010 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:23.745254 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:23.745262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:23.746755 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:23.746758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:23.747839 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:23.747850 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:23.748912 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:23.748917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:23.749033 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:23.749660 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:23.749682 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:23.750986 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:23.754046 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:23.754054 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:23.754060 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:23.754705 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:23.755135 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:23.755140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:23.755144 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:23.755324 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:23.756118 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:23.756125 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:23.756309 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 02:04:23.756418 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 02:04:23.756635 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":344,"threads_started":1}
I20260928 02:04:23.757337 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:23.743294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:49119 (local address 127.1.182.190:39221)
0928 02:04:23.743488 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:23.743496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:23.744019 (+ 523us) server_negotiation.cc:407] Connection header received
0928 02:04:23.744216 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:23.744222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:23.744308 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:23.744438 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:23.745398 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:23.746615 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:23.747976 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:23.748713 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:23.748867 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:23.751285 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:23.751290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:23.751293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:23.751322 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:23.753921 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:23.754850 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:23.754854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:23.754856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:23.754971 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:23.755467 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:23.755475 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:23.755481 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:23.755985 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:23.756147 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:23.756581 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 02:04:23.756862 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:04:23.771900 3090 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:33097
I20260928 02:04:23.771919 3224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:33097 every 8 connection(s)
I20260928 02:04:23.773203 3090 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:04:23.775305 3225 heartbeater.cc:347] Connected to a master server at 127.1.182.190:39221
I20260928 02:04:23.775555 3225 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:23.776216 3225 heartbeater.cc:511] Master 127.1.182.190:39221 requested a full tablet report, sending...
I20260928 02:04:23.777554 2879 ts_manager.cc:194] Registered new tserver with Master: 9f27378c91b242c4913a2035f885ce34 (127.1.182.130:33097)
I20260928 02:04:23.778400 2879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:47489
I20260928 02:04:23.781265 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 3090
I20260928 02:04:23.781450 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:04:23.781900 1754 external_mini_cluster.cc:1428] Setting key 7f50497d3486de2a11bdcf6c9388042b
WARNING: no policy specified for kudu/127.1.182.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
I20260928 02:04:23.822655 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39221
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:24.139173 3229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:24.139503 3229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:24.139670 3229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:24.147176 3229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:24.147310 3229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:24.147462 3229 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:04:24.147598 3229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:04:24.157397 3229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.182.190:39221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:24.158649 3229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:24.160017 3229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:24.172446 3234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:24.172482 3235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:24.172611 3237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:24.173625 3229 server_base.cc:1034] running on GCE node
I20260928 02:04:24.174235 3229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:24.175206 3229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:24.176441 3229 hybrid_clock.cc:648] HybridClock initialized: now 1790561064176380 us; error 40 us; skew 500 ppm
I20260928 02:04:24.176736 3229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:24 dist-test-slave-23bp krb5kdc[2834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:24.180136 3229 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:04:24.181689 3229 webserver.cc:460] Webserver started at http://127.1.182.131:34799/ using document root <none> and password file <none>
I20260928 02:04:24.182413 3229 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:24.182535 3229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:24.182857 3229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:24.185693 3229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "631f1ca6a4874be791cf70010b209203"
format_stamp: "Formatted at 2026-09-28 02:04:24 on dist-test-slave-23bp"
server_key: "359f707b2bef7ce0a47a294841a04fd0"
server_key_iv: "acd98befc9d6455575e8113af5ff0fe1"
server_key_version: "encryptionkey@0"
I20260928 02:04:24.186452 3229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "631f1ca6a4874be791cf70010b209203"
format_stamp: "Formatted at 2026-09-28 02:04:24 on dist-test-slave-23bp"
server_key: "359f707b2bef7ce0a47a294841a04fd0"
server_key_iv: "acd98befc9d6455575e8113af5ff0fe1"
server_key_version: "encryptionkey@0"
I20260928 02:04:24.191381 3229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:04:24.194571 3244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:24.195914 3229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:04:24.196084 3229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "631f1ca6a4874be791cf70010b209203"
format_stamp: "Formatted at 2026-09-28 02:04:24 on dist-test-slave-23bp"
server_key: "359f707b2bef7ce0a47a294841a04fd0"
server_key_iv: "acd98befc9d6455575e8113af5ff0fe1"
server_key_version: "encryptionkey@0"
I20260928 02:04:24.196290 3229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:24.222489 3229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:24.226387 3229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:24.226686 3229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:24.229310 3229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:24.229411 3229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:24.229549 3229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:24.229641 3229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:04:24 dist-test-slave-23bp krb5kdc[2834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:24.256629 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.229987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:40189 (local address 127.1.182.190:39221)
0928 02:04:24.230173 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:24.230180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:24.231687 (+ 1507us) server_negotiation.cc:407] Connection header received
0928 02:04:24.233680 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:24.233694 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:24.233782 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:24.233903 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:24.236251 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.236987 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.239013 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.239653 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.239813 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.248110 (+ 8297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:24.248120 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:24.248128 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:24.248183 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:24.252616 (+ 4433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:24.253357 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:24.253366 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:24.253371 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:24.253554 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:24.254130 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:24.254136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:24.254141 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:24.254555 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:24.254722 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:24.255794 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0928 02:04:24.256107 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:04:24.257189 3253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.230523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39221 (local address 127.1.182.131:40189)
0928 02:04:24.231503 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:24.231552 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:24.233404 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:24.234156 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:24.234170 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:24.235221 (+ 1051us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:24.236061 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:24.236083 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.237158 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.237166 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:24.238846 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:24.238857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.239968 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.239979 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:24.240360 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.241245 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:24.241284 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:24.247831 (+ 6547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:24.252775 (+ 4944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:24.252782 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:24.252794 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:24.253224 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:24.253703 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:24.253713 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:24.253719 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:24.254006 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:24.254705 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:24.254713 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:24.255264 (+ 551us) client_negotiation.cc:769] Sending connection context
0928 02:04:24.255592 (+ 328us) client_negotiation.cc:240] Negotiation successful
0928 02:04:24.256052 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":374,"threads_started":1}
I20260928 02:04:24.258992 3086 heartbeater.cc:503] Master 127.1.182.190:39221 was elected leader, sending a full tablet report...
I20260928 02:04:24.277387 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.263562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:59047 (local address 127.1.182.190:39221)
0928 02:04:24.263769 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:24.263777 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:24.264569 (+ 792us) server_negotiation.cc:407] Connection header received
0928 02:04:24.264895 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:24.264900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:24.265022 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:24.265117 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:24.266135 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.266804 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.268886 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.269495 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.269628 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.272044 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:24.272050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:24.272055 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:24.272089 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:24.274802 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:24.275536 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:24.275541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:24.275545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:24.275663 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:24.276016 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:24.276021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:24.276024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:24.276280 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:24.276426 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:24.276821 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 02:04:24.277018 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:04:24.277477 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.263632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39221 (local address 127.1.182.131:59047)
0928 02:04:24.264443 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:24.264474 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:24.264637 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:24.265131 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:24.265136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:24.265678 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:24.266002 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:24.266012 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.266937 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.266943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:24.268737 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:24.268751 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.269695 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.269701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:24.269913 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.270826 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:24.270845 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:24.271870 (+ 1025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:24.274928 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:24.274933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:24.274937 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:24.275402 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:24.275766 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:24.275772 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:24.275776 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:24.275902 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:24.276392 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:24.276397 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:24.276569 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 02:04:24.276713 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 02:04:24.277018 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":410,"threads_started":1}
I20260928 02:04:24.293982 3229 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:46553
I20260928 02:04:24.294008 3363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:46553 every 8 connection(s)
I20260928 02:04:24.295282 3229 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:04:24.297500 3364 heartbeater.cc:347] Connected to a master server at 127.1.182.190:39221
I20260928 02:04:24.297781 3364 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:24.298617 3364 heartbeater.cc:511] Master 127.1.182.190:39221 requested a full tablet report, sending...
I20260928 02:04:24.299779 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 3229
I20260928 02:04:24.299973 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:04:24.300137 2879 ts_manager.cc:194] Registered new tserver with Master: 631f1ca6a4874be791cf70010b209203 (127.1.182.131:46553)
I20260928 02:04:24.300361 1754 external_mini_cluster.cc:1428] Setting key 1fb55a5101c556ca8e5003626b8a65fa
I20260928 02:04:24.301090 2879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:40189
I20260928 02:04:24.303365 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:04:24 dist-test-slave-23bp krb5kdc[2834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561064, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:04:24 dist-test-slave-23bp krb5kdc[2834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561064, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:24.348261 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.335233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57380 (local address 127.1.182.190:39221)
0928 02:04:24.335482 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:24.335489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:24.335735 (+ 246us) server_negotiation.cc:407] Connection header received
0928 02:04:24.335912 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:24.335916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:24.335978 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:24.336064 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:24.337391 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.338094 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.339485 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.339882 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.339977 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.343763 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:24.343768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:24.343771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:24.343803 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:24.346088 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:24.346642 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:24.346645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:24.346648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:24.346773 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:24.347135 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:24.347138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:24.347140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:24.347376 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:24.347551 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:24.347830 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 02:04:24.347981 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 02:04:24.352339 2878 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:04:24.359179 2878 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:04:24.382130 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.371392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44578 (local address 127.1.182.129:40771)
0928 02:04:24.372237 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:24.372249 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:24.372704 (+ 455us) server_negotiation.cc:407] Connection header received
0928 02:04:24.373112 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:24.373122 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:24.373497 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:24.373676 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:24.374491 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.376596 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.379440 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.380940 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.381067 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.381296 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:24.381622 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 02:04:24.381782 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":551,"threads_started":1}
I20260928 02:04:24.383116 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.371178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:40771 (local address 127.0.0.1:44578)
0928 02:04:24.372566 (+ 1388us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:24.372596 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:24.372820 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:24.373712 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:24.373718 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:24.373746 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:24.374262 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:24.374281 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.376795 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.376803 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:24.379279 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:24.379293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.381136 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.381143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:24.381382 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.381422 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:04:24.381578 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 02:04:24.381791 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1195,"thread_start_us":1078,"threads_started":1}
I20260928 02:04:24.384711 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.372856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:33097 (local address 127.0.0.1:57950)
0928 02:04:24.373431 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:24.373449 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:24.373551 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:24.378109 (+ 4558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:24.378114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:24.378138 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:24.378565 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:24.378576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.380255 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.380263 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:24.382197 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:24.382211 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.383929 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.383935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:24.384084 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.384120 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:04:24.384230 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:04:24.384358 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":1,"thread_start_us":297,"threads_started":1}
I20260928 02:04:24.384724 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.373212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57950 (local address 127.1.182.130:33097)
0928 02:04:24.377541 (+ 4329us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:24.377549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:24.377575 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:04:24.377680 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:24.377686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:24.377899 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:24.378769 (+ 870us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:24.378942 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.380131 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.382378 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.383741 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.383888 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.384078 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:24.384289 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 02:04:24.384432 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4177,"thread_start_us":4052,"threads_started":1}
I20260928 02:04:24.385999 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.373502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:46553 (local address 127.0.0.1:57078)
0928 02:04:24.374287 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:24.374304 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:24.374469 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:24.378253 (+ 3784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:24.378257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:24.378283 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:24.378901 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:24.378920 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.380955 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.380961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:24.383195 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:24.383211 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.385340 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.385346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:24.385520 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.385557 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:04:24.385653 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 02:04:24.385763 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":2,"thread_start_us":874,"threads_started":1}
I20260928 02:04:24.386665 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.374162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57078 (local address 127.1.182.131:46553)
0928 02:04:24.377385 (+ 3223us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:24.377393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:24.377422 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:04:24.377516 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:24.377520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:24.377734 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:24.377919 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:24.379123 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.380810 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.383613 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.384966 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.385097 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.385314 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:24.386321 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0928 02:04:24.386442 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3043,"thread_start_us":2820,"threads_started":1}
I20260928 02:04:24.388047 3155 tablet_service.cc:1467] Processing CreateTablet for tablet 7693477c75214834a75aee4a55440423 (DEFAULT_TABLE table=test-table [id=496017b8139f4b99aa310f606c6d8646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:24.387925 3016 tablet_service.cc:1467] Processing CreateTablet for tablet 7693477c75214834a75aee4a55440423 (DEFAULT_TABLE table=test-table [id=496017b8139f4b99aa310f606c6d8646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:24.389385 3016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7693477c75214834a75aee4a55440423. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:24.389392 3155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7693477c75214834a75aee4a55440423. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:24.389351 3299 tablet_service.cc:1467] Processing CreateTablet for tablet 7693477c75214834a75aee4a55440423 (DEFAULT_TABLE table=test-table [id=496017b8139f4b99aa310f606c6d8646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:24.390437 3299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7693477c75214834a75aee4a55440423. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:24.402515 3383 tablet_bootstrap.cc:492] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203: Bootstrap starting.
I20260928 02:04:24.404515 3384 tablet_bootstrap.cc:492] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589: Bootstrap starting.
I20260928 02:04:24.405224 3385 tablet_bootstrap.cc:492] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34: Bootstrap starting.
I20260928 02:04:24.406239 3383 tablet_bootstrap.cc:654] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:24.407545 3383 log.cc:824] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:24.408565 3385 tablet_bootstrap.cc:654] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:24.409564 3385 log.cc:824] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:24.410559 3384 tablet_bootstrap.cc:654] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:24.411546 3384 log.cc:824] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:24.411626 3383 tablet_bootstrap.cc:492] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203: No bootstrap required, opened a new log
I20260928 02:04:24.412009 3383 ts_tablet_manager.cc:1397] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260928 02:04:24.413792 3384 tablet_bootstrap.cc:492] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589: No bootstrap required, opened a new log
I20260928 02:04:24.414031 3384 ts_tablet_manager.cc:1397] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 02:04:24.417532 3385 tablet_bootstrap.cc:492] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34: No bootstrap required, opened a new log
I20260928 02:04:24.417779 3385 ts_tablet_manager.cc:1397] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260928 02:04:24.422309 3384 raft_consensus.cc:348] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.422676 3384 raft_consensus.cc:374] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:24.422813 3384 raft_consensus.cc:729] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d757d0b10c450b8b6690168898b589, State: Initialized, Role: FOLLOWER
I20260928 02:04:24.423022 3383 raft_consensus.cc:348] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.423445 3383 raft_consensus.cc:374] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:24.423544 3383 raft_consensus.cc:729] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 631f1ca6a4874be791cf70010b209203, State: Initialized, Role: FOLLOWER
I20260928 02:04:24.423645 3384 consensus_queue.cc:260] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.424101 3383 consensus_queue.cc:260] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.425417 3384 ts_tablet_manager.cc:1428] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260928 02:04:24.427757 3364 heartbeater.cc:503] Master 127.1.182.190:39221 was elected leader, sending a full tablet report...
I20260928 02:04:24.428363 3383 ts_tablet_manager.cc:1428] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260928 02:04:24.429286 3385 raft_consensus.cc:348] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.429695 3385 raft_consensus.cc:374] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:24.429817 3385 raft_consensus.cc:729] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f27378c91b242c4913a2035f885ce34, State: Initialized, Role: FOLLOWER
I20260928 02:04:24.430408 3385 consensus_queue.cc:260] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.431617 3225 heartbeater.cc:503] Master 127.1.182.190:39221 was elected leader, sending a full tablet report...
I20260928 02:04:24.432076 3385 ts_tablet_manager.cc:1428] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34: Time spent starting tablet: real 0.014s user 0.011s sys 0.005s
I20260928 02:04:24.489617 3390 raft_consensus.cc:482] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:24.489988 3390 raft_consensus.cc:504] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.491729 3390 leader_election.cc:288] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95d757d0b10c450b8b6690168898b589 (127.1.182.129:40771), 9f27378c91b242c4913a2035f885ce34 (127.1.182.130:33097)
I20260928 02:04:24.498437 3253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.491989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:40771 (local address 127.1.182.131:47565)
0928 02:04:24.492229 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:24.492249 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:24.492392 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:24.492875 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:24.492879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:24.492907 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:24.493151 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:24.493165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.494485 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.494489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:24.496692 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:24.496705 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.497928 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.497932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:24.498066 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.498089 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:04:24.498159 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 02:04:24.498251 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 02:04:24.498584 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.492249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:47565 (local address 127.1.182.129:40771)
0928 02:04:24.492430 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:24.492437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:24.492459 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:04:24.492539 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:24.492546 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:24.492644 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:24.492763 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:24.493340 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.494352 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.496837 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.497797 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.497947 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.498029 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:24.498220 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 02:04:24.498352 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:04:24.499536 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.492329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:33097 (local address 127.1.182.131:33989)
0928 02:04:24.492990 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:24.493028 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:24.493170 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:24.493533 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:24.493537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:24.493559 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:24.493936 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:24.493948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.495046 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.495052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:24.497207 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:24.497216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.498917 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.498923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:24.499115 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.499151 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:04:24.499232 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:04:24.499310 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":314,"threads_started":1}
I20260928 02:04:24.499753 3036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7693477c75214834a75aee4a55440423" candidate_uuid: "631f1ca6a4874be791cf70010b209203" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95d757d0b10c450b8b6690168898b589" is_pre_election: true
I20260928 02:04:24.499974 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.492459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:33989 (local address 127.1.182.130:33097)
0928 02:04:24.492682 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:24.492688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:24.493134 (+ 446us) server_negotiation.cc:407] Connection header received
0928 02:04:24.493316 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:24.493321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:24.493399 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:24.493483 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:24.494096 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.494916 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.497358 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.498507 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.498799 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.498892 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:24.499612 (+ 720us) server_negotiation.cc:299] Negotiation successful
0928 02:04:24.499740 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:04:24.500294 3036 raft_consensus.cc:2454] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 631f1ca6a4874be791cf70010b209203 in term 0.
I20260928 02:04:24.500860 3180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7693477c75214834a75aee4a55440423" candidate_uuid: "631f1ca6a4874be791cf70010b209203" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f27378c91b242c4913a2035f885ce34" is_pre_election: true
I20260928 02:04:24.500995 3246 leader_election.cc:302] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 631f1ca6a4874be791cf70010b209203, 95d757d0b10c450b8b6690168898b589; no voters:
W20260928 02:04:24.501369 3087 tablet.cc:2369] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:04:24.501322 3180 raft_consensus.cc:2454] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 631f1ca6a4874be791cf70010b209203 in term 0.
I20260928 02:04:24.501437 3390 raft_consensus.cc:2781] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:04:24.501642 3390 raft_consensus.cc:482] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:04:24.501750 3390 raft_consensus.cc:3037] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:24.503983 3390 raft_consensus.cc:504] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.504503 3390 leader_election.cc:288] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 [CANDIDATE]: Term 1 election: Requested vote from peers 95d757d0b10c450b8b6690168898b589 (127.1.182.129:40771), 9f27378c91b242c4913a2035f885ce34 (127.1.182.130:33097)
I20260928 02:04:24.504938 3036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7693477c75214834a75aee4a55440423" candidate_uuid: "631f1ca6a4874be791cf70010b209203" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95d757d0b10c450b8b6690168898b589"
I20260928 02:04:24.505013 3180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7693477c75214834a75aee4a55440423" candidate_uuid: "631f1ca6a4874be791cf70010b209203" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f27378c91b242c4913a2035f885ce34"
I20260928 02:04:24.505131 3036 raft_consensus.cc:3037] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:24.505257 3180 raft_consensus.cc:3037] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:24.507448 3180 raft_consensus.cc:2454] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 631f1ca6a4874be791cf70010b209203 in term 1.
I20260928 02:04:24.507759 3246 leader_election.cc:302] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 631f1ca6a4874be791cf70010b209203, 9f27378c91b242c4913a2035f885ce34; no voters:
I20260928 02:04:24.507848 3036 raft_consensus.cc:2454] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 631f1ca6a4874be791cf70010b209203 in term 1.
I20260928 02:04:24.507997 3390 raft_consensus.cc:2781] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:24.508579 3390 raft_consensus.cc:686] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 [term 1 LEADER]: Becoming Leader. State: Replica: 631f1ca6a4874be791cf70010b209203, State: Running, Role: LEADER
I20260928 02:04:24.509018 3390 consensus_queue.cc:237] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.513623 2878 catalog_manager.cc:5495] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 reported cstate change: term changed from 0 to 1, leader changed from <none> to 631f1ca6a4874be791cf70010b209203 (127.1.182.131). New cstate: current_term: 1 leader_uuid: "631f1ca6a4874be791cf70010b209203" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } health_report { overall_health: HEALTHY } } }
W20260928 02:04:24.525262 3226 tablet.cc:2369] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:04:24.541630 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.536087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57080 (local address 127.1.182.131:46553)
0928 02:04:24.536294 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:24.536299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:24.536446 (+ 147us) server_negotiation.cc:407] Connection header received
0928 02:04:24.536770 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:24.536774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:24.536847 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:24.536988 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:04:24.537371 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.538033 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.539444 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.539855 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.539991 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.540480 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:04:24.541002 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:04:24.541161 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:24.541303 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 02:04:24.541381 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 02:04:24.546953 3365 tablet.cc:2369] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:04:24.550166 3180 raft_consensus.cc:1261] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34 [term 1 FOLLOWER]: Refusing update from remote peer 631f1ca6a4874be791cf70010b209203: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:04:24.550165 3036 raft_consensus.cc:1261] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589 [term 1 FOLLOWER]: Refusing update from remote peer 631f1ca6a4874be791cf70010b209203: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:04:24.551209 3393 consensus_queue.cc:1035] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:24.551558 3390 consensus_queue.cc:1035] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:24.580623 2879 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:04:24.580895 2879 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:04:24.584443 2879 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229: Sending DeleteTablet for 3 replicas of tablet 7693477c75214834a75aee4a55440423
I20260928 02:04:24.585512 3016 tablet_service.cc:1514] Processing DeleteTablet for tablet 7693477c75214834a75aee4a55440423 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:24 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:44578
I20260928 02:04:24.585654 3155 tablet_service.cc:1514] Processing DeleteTablet for tablet 7693477c75214834a75aee4a55440423 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:24 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:57950
I20260928 02:04:24.586299 3299 tablet_service.cc:1514] Processing DeleteTablet for tablet 7693477c75214834a75aee4a55440423 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:24 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:57078
I20260928 02:04:24.587786 2879 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:04:24.588667 2879 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:04:24.591377 3407 tablet_replica.cc:330] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34: stopping tablet replica
I20260928 02:04:24.591831 3407 raft_consensus.cc:2229] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:04:24.592788 3407 raft_consensus.cc:2258] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:24.593523 3406 tablet_replica.cc:330] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589: stopping tablet replica
I20260928 02:04:24.594085 3406 raft_consensus.cc:2229] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:04:24.595633 3406 raft_consensus.cc:2258] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:24.597733 3407 ts_tablet_manager.cc:1905] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:24.597848 3298 tablet_service.cc:1467] Processing CreateTablet for tablet ea35626beb4844c09b8eefbf89815745 (DEFAULT_TABLE table=test-table [id=1efe6c6f4ef04f1a9c0dd35b922aceb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:24.598469 3298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea35626beb4844c09b8eefbf89815745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:24.603583 3408 tablet_replica.cc:330] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203: stopping tablet replica
I20260928 02:04:24.603744 3016 tablet_service.cc:1467] Processing CreateTablet for tablet ea35626beb4844c09b8eefbf89815745 (DEFAULT_TABLE table=test-table [id=1efe6c6f4ef04f1a9c0dd35b922aceb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:24.604110 3408 raft_consensus.cc:2229] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:04:24.604260 3016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea35626beb4844c09b8eefbf89815745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:24.604931 3408 raft_consensus.cc:2258] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:24.607111 3407 ts_tablet_manager.cc:1918] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:04:24.607334 3407 log.cc:1196] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/7693477c75214834a75aee4a55440423
I20260928 02:04:24.607921 3407 ts_tablet_manager.cc:1939] T 7693477c75214834a75aee4a55440423 P 9f27378c91b242c4913a2035f885ce34: Deleting consensus metadata
I20260928 02:04:24.609773 2864 catalog_manager.cc:4841] TS 9f27378c91b242c4913a2035f885ce34 (127.1.182.130:33097): tablet 7693477c75214834a75aee4a55440423 (table test-table [id=496017b8139f4b99aa310f606c6d8646]) successfully deleted
I20260928 02:04:24.612779 3384 tablet_bootstrap.cc:492] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589: Bootstrap starting.
I20260928 02:04:24.612937 3383 tablet_bootstrap.cc:492] T ea35626beb4844c09b8eefbf89815745 P 631f1ca6a4874be791cf70010b209203: Bootstrap starting.
I20260928 02:04:24.613221 3155 tablet_service.cc:1467] Processing CreateTablet for tablet ea35626beb4844c09b8eefbf89815745 (DEFAULT_TABLE table=test-table [id=1efe6c6f4ef04f1a9c0dd35b922aceb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:24.613616 3155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea35626beb4844c09b8eefbf89815745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:24.614154 3408 ts_tablet_manager.cc:1905] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:24.614399 3384 tablet_bootstrap.cc:654] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:24.614542 3383 tablet_bootstrap.cc:654] T ea35626beb4844c09b8eefbf89815745 P 631f1ca6a4874be791cf70010b209203: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:24.620716 3385 tablet_bootstrap.cc:492] T ea35626beb4844c09b8eefbf89815745 P 9f27378c91b242c4913a2035f885ce34: Bootstrap starting.
I20260928 02:04:24.622253 3385 tablet_bootstrap.cc:654] T ea35626beb4844c09b8eefbf89815745 P 9f27378c91b242c4913a2035f885ce34: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:24.622434 3408 ts_tablet_manager.cc:1918] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:04:24.622903 3408 log.cc:1196] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/7693477c75214834a75aee4a55440423
I20260928 02:04:24.623517 3408 ts_tablet_manager.cc:1939] T 7693477c75214834a75aee4a55440423 P 631f1ca6a4874be791cf70010b209203: Deleting consensus metadata
I20260928 02:04:24.624018 3384 tablet_bootstrap.cc:492] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589: No bootstrap required, opened a new log
I20260928 02:04:24.624171 3384 ts_tablet_manager.cc:1397] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260928 02:04:24.625413 3384 raft_consensus.cc:348] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.625638 3384 raft_consensus.cc:374] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:24.625734 3384 raft_consensus.cc:729] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d757d0b10c450b8b6690168898b589, State: Initialized, Role: FOLLOWER
I20260928 02:04:24.625986 3384 consensus_queue.cc:260] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.626675 3384 ts_tablet_manager.cc:1428] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:04:24.626632 2864 catalog_manager.cc:4841] TS 631f1ca6a4874be791cf70010b209203 (127.1.182.131:46553): tablet 7693477c75214834a75aee4a55440423 (table test-table [id=496017b8139f4b99aa310f606c6d8646]) successfully deleted
I20260928 02:04:24.626960 3383 tablet_bootstrap.cc:492] T ea35626beb4844c09b8eefbf89815745 P 631f1ca6a4874be791cf70010b209203: No bootstrap required, opened a new log
I20260928 02:04:24.627619 3383 ts_tablet_manager.cc:1397] T ea35626beb4844c09b8eefbf89815745 P 631f1ca6a4874be791cf70010b209203: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.001s
I20260928 02:04:24.627679 3406 ts_tablet_manager.cc:1905] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:24.628671 3383 raft_consensus.cc:348] T ea35626beb4844c09b8eefbf89815745 P 631f1ca6a4874be791cf70010b209203 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.628962 3383 raft_consensus.cc:374] T ea35626beb4844c09b8eefbf89815745 P 631f1ca6a4874be791cf70010b209203 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:24.629078 3383 raft_consensus.cc:729] T ea35626beb4844c09b8eefbf89815745 P 631f1ca6a4874be791cf70010b209203 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 631f1ca6a4874be791cf70010b209203, State: Initialized, Role: FOLLOWER
I20260928 02:04:24.629383 3383 consensus_queue.cc:260] T ea35626beb4844c09b8eefbf89815745 P 631f1ca6a4874be791cf70010b209203 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.633423 3406 ts_tablet_manager.cc:1918] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:04:24.633595 3406 log.cc:1196] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-0/wal/wals/7693477c75214834a75aee4a55440423
I20260928 02:04:24.634375 3383 ts_tablet_manager.cc:1428] T ea35626beb4844c09b8eefbf89815745 P 631f1ca6a4874be791cf70010b209203: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260928 02:04:24.639509 3406 ts_tablet_manager.cc:1939] T 7693477c75214834a75aee4a55440423 P 95d757d0b10c450b8b6690168898b589: Deleting consensus metadata
I20260928 02:04:24.641052 2864 catalog_manager.cc:4841] TS 95d757d0b10c450b8b6690168898b589 (127.1.182.129:40771): tablet 7693477c75214834a75aee4a55440423 (table test-table [id=496017b8139f4b99aa310f606c6d8646]) successfully deleted
I20260928 02:04:24.646270 3385 tablet_bootstrap.cc:492] T ea35626beb4844c09b8eefbf89815745 P 9f27378c91b242c4913a2035f885ce34: No bootstrap required, opened a new log
I20260928 02:04:24.646430 3385 ts_tablet_manager.cc:1397] T ea35626beb4844c09b8eefbf89815745 P 9f27378c91b242c4913a2035f885ce34: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.011s
I20260928 02:04:24.647276 3385 raft_consensus.cc:348] T ea35626beb4844c09b8eefbf89815745 P 9f27378c91b242c4913a2035f885ce34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.647554 3385 raft_consensus.cc:374] T ea35626beb4844c09b8eefbf89815745 P 9f27378c91b242c4913a2035f885ce34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:24.647657 3385 raft_consensus.cc:729] T ea35626beb4844c09b8eefbf89815745 P 9f27378c91b242c4913a2035f885ce34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f27378c91b242c4913a2035f885ce34, State: Initialized, Role: FOLLOWER
I20260928 02:04:24.647933 3385 consensus_queue.cc:260] T ea35626beb4844c09b8eefbf89815745 P 9f27378c91b242c4913a2035f885ce34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.649384 3385 ts_tablet_manager.cc:1428] T ea35626beb4844c09b8eefbf89815745 P 9f27378c91b242c4913a2035f885ce34: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:04:24.746771 3389 raft_consensus.cc:482] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:24.747009 3389 raft_consensus.cc:504] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.748668 3389 leader_election.cc:288] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f27378c91b242c4913a2035f885ce34 (127.1.182.130:33097), 631f1ca6a4874be791cf70010b209203 (127.1.182.131:46553)
I20260928 02:04:24.757129 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.749777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:48897 (local address 127.1.182.131:46553)
0928 02:04:24.749979 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:24.749986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:24.750363 (+ 377us) server_negotiation.cc:407] Connection header received
0928 02:04:24.750615 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:24.750618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:24.750672 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:24.750794 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:24.751673 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.752632 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.754565 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.756099 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.756253 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.756334 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:24.756705 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 02:04:24.756827 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:04:24.757141 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.749003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:51011 (local address 127.1.182.130:33097)
0928 02:04:24.749258 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:24.749264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:24.749616 (+ 352us) server_negotiation.cc:407] Connection header received
0928 02:04:24.749899 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:24.749906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:24.749978 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:24.750037 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:24.750812 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.751876 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.754559 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.755829 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.755961 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.756034 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:24.756711 (+ 677us) server_negotiation.cc:299] Negotiation successful
0928 02:04:24.756847 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:04:24.757462 3413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.749626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:46553 (local address 127.1.182.129:48897)
0928 02:04:24.750266 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:24.750279 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:24.750442 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:24.751170 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:24.751175 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:24.751200 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:24.751529 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:24.751539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.752757 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.752768 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:24.754363 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:24.754382 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.756316 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.756322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:24.756501 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.756560 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 02:04:24.756666 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:04:24.756790 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":84,"thread_start_us":352,"threads_started":1}
I20260928 02:04:24.758250 3319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea35626beb4844c09b8eefbf89815745" candidate_uuid: "95d757d0b10c450b8b6690168898b589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "631f1ca6a4874be791cf70010b209203" is_pre_election: true
I20260928 02:04:24.758493 3412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.748913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:33097 (local address 127.1.182.129:51011)
0928 02:04:24.749505 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:24.749524 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:24.749724 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:24.750265 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:24.750270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:24.750303 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:24.750632 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:24.750644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.752011 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.752016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:24.754389 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:24.754402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.756239 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.756245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:24.756448 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.756559 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 02:04:24.757368 (+ 809us) client_negotiation.cc:240] Negotiation successful
0928 02:04:24.757973 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":89,"thread_start_us":308,"threads_started":1}
I20260928 02:04:24.758697 3319 raft_consensus.cc:2454] T ea35626beb4844c09b8eefbf89815745 P 631f1ca6a4874be791cf70010b209203 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95d757d0b10c450b8b6690168898b589 in term 0.
I20260928 02:04:24.759043 3180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea35626beb4844c09b8eefbf89815745" candidate_uuid: "95d757d0b10c450b8b6690168898b589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f27378c91b242c4913a2035f885ce34" is_pre_election: true
I20260928 02:04:24.759323 3180 raft_consensus.cc:2454] T ea35626beb4844c09b8eefbf89815745 P 9f27378c91b242c4913a2035f885ce34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95d757d0b10c450b8b6690168898b589 in term 0.
I20260928 02:04:24.759330 2967 leader_election.cc:302] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 631f1ca6a4874be791cf70010b209203, 95d757d0b10c450b8b6690168898b589; no voters:
I20260928 02:04:24.759994 3389 raft_consensus.cc:2781] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:04:24.760120 3389 raft_consensus.cc:482] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:04:24.760231 3389 raft_consensus.cc:3037] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:24.762275 3389 raft_consensus.cc:504] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.762874 3389 leader_election.cc:288] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 [CANDIDATE]: Term 1 election: Requested vote from peers 9f27378c91b242c4913a2035f885ce34 (127.1.182.130:33097), 631f1ca6a4874be791cf70010b209203 (127.1.182.131:46553)
I20260928 02:04:24.763324 3180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea35626beb4844c09b8eefbf89815745" candidate_uuid: "95d757d0b10c450b8b6690168898b589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f27378c91b242c4913a2035f885ce34"
I20260928 02:04:24.763448 3319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea35626beb4844c09b8eefbf89815745" candidate_uuid: "95d757d0b10c450b8b6690168898b589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "631f1ca6a4874be791cf70010b209203"
I20260928 02:04:24.763526 3180 raft_consensus.cc:3037] T ea35626beb4844c09b8eefbf89815745 P 9f27378c91b242c4913a2035f885ce34 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:24.763662 3319 raft_consensus.cc:3037] T ea35626beb4844c09b8eefbf89815745 P 631f1ca6a4874be791cf70010b209203 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:24.766026 3180 raft_consensus.cc:2454] T ea35626beb4844c09b8eefbf89815745 P 9f27378c91b242c4913a2035f885ce34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95d757d0b10c450b8b6690168898b589 in term 1.
I20260928 02:04:24.766441 2967 leader_election.cc:302] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95d757d0b10c450b8b6690168898b589, 9f27378c91b242c4913a2035f885ce34; no voters:
I20260928 02:04:24.766764 3389 raft_consensus.cc:2781] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:24.767289 3319 raft_consensus.cc:2454] T ea35626beb4844c09b8eefbf89815745 P 631f1ca6a4874be791cf70010b209203 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95d757d0b10c450b8b6690168898b589 in term 1.
I20260928 02:04:24.767316 3389 raft_consensus.cc:686] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 [term 1 LEADER]: Becoming Leader. State: Replica: 95d757d0b10c450b8b6690168898b589, State: Running, Role: LEADER
I20260928 02:04:24.767874 3389 consensus_queue.cc:237] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.771268 2879 catalog_manager.cc:5495] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95d757d0b10c450b8b6690168898b589 (127.1.182.129). New cstate: current_term: 1 leader_uuid: "95d757d0b10c450b8b6690168898b589" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:04:24 dist-test-slave-23bp krb5kdc[2834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561062, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:24.836419 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.821536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57390 (local address 127.1.182.190:39221)
0928 02:04:24.821714 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:24.821719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:24.822397 (+ 678us) server_negotiation.cc:407] Connection header received
0928 02:04:24.822600 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:24.822605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:24.822688 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:24.822795 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:24.824413 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.825118 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.826576 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.827123 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.827268 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.830982 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:24.830991 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:24.831000 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:24.831027 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:24.833683 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:24.834364 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:24.834370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:24.834377 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:24.834507 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:24.834864 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:24.834869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:24.834872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:24.835200 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:24.835349 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:24.835774 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 02:04:24.836106 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":55}
I20260928 02:04:24.836460 3421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:24.821644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39221 (local address 127.0.0.1:57390)
0928 02:04:24.822292 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:24.822308 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:24.822476 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:24.822820 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:24.822824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:24.824008 (+ 1184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:24.824262 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:24.824273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.825246 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.825254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:24.826420 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:24.826435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:24.827305 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:24.827311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:24.827474 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:24.828200 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:24.828221 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:24.830800 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:24.833824 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:24.833837 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:24.833846 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:24.834246 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:24.834610 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:24.834617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:24.834620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:24.834752 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:24.835326 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:24.835333 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:24.835508 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 02:04:24.835687 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 02:04:24.835991 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":333,"threads_started":1}
I20260928 02:04:24.843189 2879 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:57390:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:04:24.844506 2879 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:04:24.851681 3016 tablet_service.cc:1467] Processing CreateTablet for tablet 629eae93873842da8c3e24fe3013a0fe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9b506cad3a654829a53b1380d3f3e5b2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:04:24.852276 3016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 629eae93873842da8c3e24fe3013a0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:24.852246 3298 tablet_service.cc:1467] Processing CreateTablet for tablet 629eae93873842da8c3e24fe3013a0fe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9b506cad3a654829a53b1380d3f3e5b2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:04:24.852777 3298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 629eae93873842da8c3e24fe3013a0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:24.855633 3155 tablet_service.cc:1467] Processing CreateTablet for tablet 629eae93873842da8c3e24fe3013a0fe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9b506cad3a654829a53b1380d3f3e5b2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:04:24.856172 3155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 629eae93873842da8c3e24fe3013a0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:24.859589 3383 tablet_bootstrap.cc:492] T 629eae93873842da8c3e24fe3013a0fe P 631f1ca6a4874be791cf70010b209203: Bootstrap starting.
I20260928 02:04:24.859786 3384 tablet_bootstrap.cc:492] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589: Bootstrap starting.
I20260928 02:04:24.861369 3383 tablet_bootstrap.cc:654] T 629eae93873842da8c3e24fe3013a0fe P 631f1ca6a4874be791cf70010b209203: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:24.861371 3384 tablet_bootstrap.cc:654] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:24.865499 3385 tablet_bootstrap.cc:492] T 629eae93873842da8c3e24fe3013a0fe P 9f27378c91b242c4913a2035f885ce34: Bootstrap starting.
I20260928 02:04:24.867166 3385 tablet_bootstrap.cc:654] T 629eae93873842da8c3e24fe3013a0fe P 9f27378c91b242c4913a2035f885ce34: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:24.871342 3385 tablet_bootstrap.cc:492] T 629eae93873842da8c3e24fe3013a0fe P 9f27378c91b242c4913a2035f885ce34: No bootstrap required, opened a new log
I20260928 02:04:24.871816 3385 ts_tablet_manager.cc:1397] T 629eae93873842da8c3e24fe3013a0fe P 9f27378c91b242c4913a2035f885ce34: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 02:04:24.872745 3385 raft_consensus.cc:348] T 629eae93873842da8c3e24fe3013a0fe P 9f27378c91b242c4913a2035f885ce34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.873838 3383 tablet_bootstrap.cc:492] T 629eae93873842da8c3e24fe3013a0fe P 631f1ca6a4874be791cf70010b209203: No bootstrap required, opened a new log
I20260928 02:04:24.874022 3383 ts_tablet_manager.cc:1397] T 629eae93873842da8c3e24fe3013a0fe P 631f1ca6a4874be791cf70010b209203: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260928 02:04:24.874152 3385 raft_consensus.cc:374] T 629eae93873842da8c3e24fe3013a0fe P 9f27378c91b242c4913a2035f885ce34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:24.874302 3385 raft_consensus.cc:729] T 629eae93873842da8c3e24fe3013a0fe P 9f27378c91b242c4913a2035f885ce34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f27378c91b242c4913a2035f885ce34, State: Initialized, Role: FOLLOWER
I20260928 02:04:24.874611 3385 consensus_queue.cc:260] T 629eae93873842da8c3e24fe3013a0fe P 9f27378c91b242c4913a2035f885ce34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.874946 3383 raft_consensus.cc:348] T 629eae93873842da8c3e24fe3013a0fe P 631f1ca6a4874be791cf70010b209203 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.875216 3383 raft_consensus.cc:374] T 629eae93873842da8c3e24fe3013a0fe P 631f1ca6a4874be791cf70010b209203 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:24.875296 3383 raft_consensus.cc:729] T 629eae93873842da8c3e24fe3013a0fe P 631f1ca6a4874be791cf70010b209203 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 631f1ca6a4874be791cf70010b209203, State: Initialized, Role: FOLLOWER
I20260928 02:04:24.875169 3391 tablet_replica.cc:439] T 629eae93873842da8c3e24fe3013a0fe P 9f27378c91b242c4913a2035f885ce34: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } } }
I20260928 02:04:24.875560 3391 tablet_replica.cc:442] T 629eae93873842da8c3e24fe3013a0fe P 9f27378c91b242c4913a2035f885ce34: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:04:24.875679 3385 ts_tablet_manager.cc:1428] T 629eae93873842da8c3e24fe3013a0fe P 9f27378c91b242c4913a2035f885ce34: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260928 02:04:24.875581 3383 consensus_queue.cc:260] T 629eae93873842da8c3e24fe3013a0fe P 631f1ca6a4874be791cf70010b209203 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.876047 3401 tablet_replica.cc:439] T 629eae93873842da8c3e24fe3013a0fe P 631f1ca6a4874be791cf70010b209203: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } } }
I20260928 02:04:24.876372 3401 tablet_replica.cc:442] T 629eae93873842da8c3e24fe3013a0fe P 631f1ca6a4874be791cf70010b209203: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:04:24.877954 3384 tablet_bootstrap.cc:492] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589: No bootstrap required, opened a new log
I20260928 02:04:24.878118 3384 ts_tablet_manager.cc:1397] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.011s
I20260928 02:04:24.879606 3383 ts_tablet_manager.cc:1428] T 629eae93873842da8c3e24fe3013a0fe P 631f1ca6a4874be791cf70010b209203: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260928 02:04:24.879660 3384 raft_consensus.cc:348] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.879925 3384 raft_consensus.cc:374] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:24.880012 3384 raft_consensus.cc:729] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d757d0b10c450b8b6690168898b589, State: Initialized, Role: FOLLOWER
I20260928 02:04:24.880272 3384 consensus_queue.cc:260] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:24.880695 3389 tablet_replica.cc:439] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } } }
I20260928 02:04:24.880971 3389 tablet_replica.cc:442] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:04:24.881811 3384 ts_tablet_manager.cc:1428] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260928 02:04:25.084986 3389 raft_consensus.cc:482] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:25.085210 3389 raft_consensus.cc:504] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:25.085867 3389 leader_election.cc:288] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f27378c91b242c4913a2035f885ce34 (127.1.182.130:33097), 631f1ca6a4874be791cf70010b209203 (127.1.182.131:46553)
I20260928 02:04:25.086526 3180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629eae93873842da8c3e24fe3013a0fe" candidate_uuid: "95d757d0b10c450b8b6690168898b589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f27378c91b242c4913a2035f885ce34" is_pre_election: true
I20260928 02:04:25.086526 3319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629eae93873842da8c3e24fe3013a0fe" candidate_uuid: "95d757d0b10c450b8b6690168898b589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "631f1ca6a4874be791cf70010b209203" is_pre_election: true
I20260928 02:04:25.086789 3180 raft_consensus.cc:2454] T 629eae93873842da8c3e24fe3013a0fe P 9f27378c91b242c4913a2035f885ce34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95d757d0b10c450b8b6690168898b589 in term 0.
I20260928 02:04:25.086788 3319 raft_consensus.cc:2454] T 629eae93873842da8c3e24fe3013a0fe P 631f1ca6a4874be791cf70010b209203 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95d757d0b10c450b8b6690168898b589 in term 0.
I20260928 02:04:25.087256 2967 leader_election.cc:302] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95d757d0b10c450b8b6690168898b589, 9f27378c91b242c4913a2035f885ce34; no voters:
I20260928 02:04:25.087551 3389 raft_consensus.cc:2781] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:04:25.087666 3389 raft_consensus.cc:482] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:04:25.087770 3389 raft_consensus.cc:3037] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:25.089676 3389 raft_consensus.cc:504] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:25.090230 3389 leader_election.cc:288] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589 [CANDIDATE]: Term 1 election: Requested vote from peers 9f27378c91b242c4913a2035f885ce34 (127.1.182.130:33097), 631f1ca6a4874be791cf70010b209203 (127.1.182.131:46553)
I20260928 02:04:25.090607 3180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629eae93873842da8c3e24fe3013a0fe" candidate_uuid: "95d757d0b10c450b8b6690168898b589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f27378c91b242c4913a2035f885ce34"
I20260928 02:04:25.090641 3319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629eae93873842da8c3e24fe3013a0fe" candidate_uuid: "95d757d0b10c450b8b6690168898b589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "631f1ca6a4874be791cf70010b209203"
I20260928 02:04:25.090827 3180 raft_consensus.cc:3037] T 629eae93873842da8c3e24fe3013a0fe P 9f27378c91b242c4913a2035f885ce34 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:25.090849 3319 raft_consensus.cc:3037] T 629eae93873842da8c3e24fe3013a0fe P 631f1ca6a4874be791cf70010b209203 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:25.092876 3319 raft_consensus.cc:2454] T 629eae93873842da8c3e24fe3013a0fe P 631f1ca6a4874be791cf70010b209203 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95d757d0b10c450b8b6690168898b589 in term 1.
I20260928 02:04:25.092870 3180 raft_consensus.cc:2454] T 629eae93873842da8c3e24fe3013a0fe P 9f27378c91b242c4913a2035f885ce34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95d757d0b10c450b8b6690168898b589 in term 1.
I20260928 02:04:25.093304 2967 leader_election.cc:302] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95d757d0b10c450b8b6690168898b589, 9f27378c91b242c4913a2035f885ce34; no voters:
I20260928 02:04:25.093542 3389 raft_consensus.cc:2781] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:25.093726 3389 raft_consensus.cc:686] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589 [term 1 LEADER]: Becoming Leader. State: Replica: 95d757d0b10c450b8b6690168898b589, State: Running, Role: LEADER
I20260928 02:04:25.093986 3389 consensus_queue.cc:237] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } }
I20260928 02:04:25.095342 3414 tablet_replica.cc:439] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589: TxnStatusTablet state changed. Reason: New leader 95d757d0b10c450b8b6690168898b589. Latest consensus state: current_term: 1 leader_uuid: "95d757d0b10c450b8b6690168898b589" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } } }
I20260928 02:04:25.095593 3414 tablet_replica.cc:442] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:04:25.096227 3426 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:04:25.096478 3426 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:04:25.096906 2877 catalog_manager.cc:5495] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95d757d0b10c450b8b6690168898b589 (127.1.182.129). New cstate: current_term: 1 leader_uuid: "95d757d0b10c450b8b6690168898b589" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } health_report { overall_health: UNKNOWN } } }
I20260928 02:04:25.121064 3421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:25.112793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:40771 (local address 127.0.0.1:44586)
0928 02:04:25.113045 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:25.113062 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:25.113163 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:25.115853 (+ 2690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:25.115858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:25.115870 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:04:25.116086 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:25.116096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:25.117251 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:25.117259 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:25.118231 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:25.118242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:25.119004 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:25.119009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:25.119218 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:25.119275 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:04:25.120460 (+ 1185us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:04:25.120480 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 02:04:25.120596 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 02:04:25.120751 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 02:04:25.121106 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:25.112973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44586 (local address 127.1.182.129:40771)
0928 02:04:25.115479 (+ 2506us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:25.115483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:25.115502 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:04:25.115589 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:25.115592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:25.115663 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:25.115836 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:04:25.116248 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:25.117116 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:25.118380 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:25.118840 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:25.118980 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:25.119490 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:04:25.120347 (+ 857us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:04:25.120507 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:25.120659 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 02:04:25.120745 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2390,"thread_start_us":2258,"threads_started":1}
I20260928 02:04:25.125824 3180 raft_consensus.cc:1261] T 629eae93873842da8c3e24fe3013a0fe P 9f27378c91b242c4913a2035f885ce34 [term 1 FOLLOWER]: Refusing update from remote peer 95d757d0b10c450b8b6690168898b589: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:04:25.126709 3389 consensus_queue.cc:1035] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:25.126803 3319 raft_consensus.cc:1261] T 629eae93873842da8c3e24fe3013a0fe P 631f1ca6a4874be791cf70010b209203 [term 1 FOLLOWER]: Refusing update from remote peer 95d757d0b10c450b8b6690168898b589: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:04:25.127578 3414 consensus_queue.cc:1035] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:25.135917 3401 tablet_replica.cc:439] T 629eae93873842da8c3e24fe3013a0fe P 631f1ca6a4874be791cf70010b209203: TxnStatusTablet state changed. Reason: New leader 95d757d0b10c450b8b6690168898b589. Latest consensus state: current_term: 1 leader_uuid: "95d757d0b10c450b8b6690168898b589" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } } }
I20260928 02:04:25.136206 3401 tablet_replica.cc:442] T 629eae93873842da8c3e24fe3013a0fe P 631f1ca6a4874be791cf70010b209203: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:04:25.137935 3430 mvcc.cc:205] Tried to move back new op lower bound from 7334138122748071936 to 7334138122626646016. Current Snapshot: MvccSnapshot[applied={T|T < 7334138122748071936}]
I20260928 02:04:25.138470 3432 mvcc.cc:205] Tried to move back new op lower bound from 7334138122748071936 to 7334138122626646016. Current Snapshot: MvccSnapshot[applied={T|T < 7334138122748071936}]
I20260928 02:04:25.139906 3401 tablet_replica.cc:439] T 629eae93873842da8c3e24fe3013a0fe P 631f1ca6a4874be791cf70010b209203: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95d757d0b10c450b8b6690168898b589" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } } }
I20260928 02:04:25.140132 3401 tablet_replica.cc:442] T 629eae93873842da8c3e24fe3013a0fe P 631f1ca6a4874be791cf70010b209203: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:04:25.143576 3414 tablet_replica.cc:439] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95d757d0b10c450b8b6690168898b589" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } } }
I20260928 02:04:25.143895 3414 tablet_replica.cc:442] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:04:25.168076 3391 tablet_replica.cc:439] T 629eae93873842da8c3e24fe3013a0fe P 9f27378c91b242c4913a2035f885ce34: TxnStatusTablet state changed. Reason: New leader 95d757d0b10c450b8b6690168898b589. Latest consensus state: current_term: 1 leader_uuid: "95d757d0b10c450b8b6690168898b589" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } } }
I20260928 02:04:25.168413 3391 tablet_replica.cc:442] T 629eae93873842da8c3e24fe3013a0fe P 9f27378c91b242c4913a2035f885ce34: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:04:25.170258 3414 tablet_replica.cc:439] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95d757d0b10c450b8b6690168898b589" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } } }
I20260928 02:04:25.170555 3414 tablet_replica.cc:442] T 629eae93873842da8c3e24fe3013a0fe P 95d757d0b10c450b8b6690168898b589: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:04:25.173718 3391 tablet_replica.cc:439] T 629eae93873842da8c3e24fe3013a0fe P 9f27378c91b242c4913a2035f885ce34: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95d757d0b10c450b8b6690168898b589" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d757d0b10c450b8b6690168898b589" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40771 } } peers { permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 } } peers { permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 } } }
I20260928 02:04:25.173950 3391 tablet_replica.cc:442] T 629eae93873842da8c3e24fe3013a0fe P 9f27378c91b242c4913a2035f885ce34: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:04:25.176901 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:25.161174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44600 (local address 127.1.182.129:40771)
0928 02:04:25.161420 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:25.161426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:25.170231 (+ 8805us) server_negotiation.cc:407] Connection header received
0928 02:04:25.170525 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:25.170530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:25.170626 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:25.170689 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:04:25.171465 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:25.172537 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:25.174687 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:25.175418 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:25.175566 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:25.176085 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:04:25.176228 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:04:25.176298 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:25.176497 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 02:04:25.176590 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68,"spinlock_wait_cycles":3840}
I20260928 02:04:25.194348 3443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:25.187679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:40771 (local address 127.1.182.129:46787)
0928 02:04:25.188356 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:25.188372 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:25.188481 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:25.188801 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:25.188804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:25.188813 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:04:25.189111 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:25.189118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:25.190584 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:25.190587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:25.191816 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:25.191827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:25.192793 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:25.192797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:25.192910 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:25.192960 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:04:25.193372 (+ 412us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:04:25.193387 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 02:04:25.193447 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 02:04:25.193589 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":348,"threads_started":1}
I20260928 02:04:25.194378 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:25.187652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:46787 (local address 127.1.182.129:40771)
0928 02:04:25.187884 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:25.187891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:25.188448 (+ 557us) server_negotiation.cc:407] Connection header received
0928 02:04:25.188595 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:25.188601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:25.188692 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:25.188786 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:04:25.189261 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:25.190466 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:25.191944 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:25.192616 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:25.192748 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:25.193121 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:04:25.193265 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:04:25.193364 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:25.193530 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 02:04:25.193636 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":77}
I20260928 02:04:25.203390 3319 raft_consensus.cc:1261] T ea35626beb4844c09b8eefbf89815745 P 631f1ca6a4874be791cf70010b209203 [term 1 FOLLOWER]: Refusing update from remote peer 95d757d0b10c450b8b6690168898b589: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:04:25.203613 3180 raft_consensus.cc:1261] T ea35626beb4844c09b8eefbf89815745 P 9f27378c91b242c4913a2035f885ce34 [term 1 FOLLOWER]: Refusing update from remote peer 95d757d0b10c450b8b6690168898b589: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:04:25.203998 3437 consensus_queue.cc:1035] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "631f1ca6a4874be791cf70010b209203" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:25.204421 3389 consensus_queue.cc:1035] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f27378c91b242c4913a2035f885ce34" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:25.267432 2877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:04:25.267812 2877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:04:25.272567 2877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b869a603f074509bb36b8ffb3102229: Sending DeleteTablet for 3 replicas of tablet ea35626beb4844c09b8eefbf89815745
I20260928 02:04:25.273690 3155 tablet_service.cc:1514] Processing DeleteTablet for tablet ea35626beb4844c09b8eefbf89815745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:25 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:57950
I20260928 02:04:25.273731 3298 tablet_service.cc:1514] Processing DeleteTablet for tablet ea35626beb4844c09b8eefbf89815745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:25 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:57078
I20260928 02:04:25.274142 3015 tablet_service.cc:1514] Processing DeleteTablet for tablet ea35626beb4844c09b8eefbf89815745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:25 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:44578
I20260928 02:04:25.282294 3450 tablet_replica.cc:330] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589: stopping tablet replica
I20260928 02:04:25.283123 3450 raft_consensus.cc:2229] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:04:25.284005 3450 raft_consensus.cc:2258] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:25.284651 3448 tablet_replica.cc:330] T ea35626beb4844c09b8eefbf89815745 P 9f27378c91b242c4913a2035f885ce34: stopping tablet replica
I20260928 02:04:25.285307 3448 raft_consensus.cc:2229] T ea35626beb4844c09b8eefbf89815745 P 9f27378c91b242c4913a2035f885ce34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:04:25.286442 3450 ts_tablet_manager.cc:1905] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:25.286811 3448 raft_consensus.cc:2258] T ea35626beb4844c09b8eefbf89815745 P 9f27378c91b242c4913a2035f885ce34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:25.287465 3449 tablet_replica.cc:330] T ea35626beb4844c09b8eefbf89815745 P 631f1ca6a4874be791cf70010b209203: stopping tablet replica
I20260928 02:04:25.287683 3449 raft_consensus.cc:2229] T ea35626beb4844c09b8eefbf89815745 P 631f1ca6a4874be791cf70010b209203 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:04:25.287920 3449 raft_consensus.cc:2258] T ea35626beb4844c09b8eefbf89815745 P 631f1ca6a4874be791cf70010b209203 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:25.288825 3448 ts_tablet_manager.cc:1905] T ea35626beb4844c09b8eefbf89815745 P 9f27378c91b242c4913a2035f885ce34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:25.289160 3449 ts_tablet_manager.cc:1905] T ea35626beb4844c09b8eefbf89815745 P 631f1ca6a4874be791cf70010b209203: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:25.297318 3449 ts_tablet_manager.cc:1918] T ea35626beb4844c09b8eefbf89815745 P 631f1ca6a4874be791cf70010b209203: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:04:25.300817 3449 log.cc:1196] T ea35626beb4844c09b8eefbf89815745 P 631f1ca6a4874be791cf70010b209203: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/ea35626beb4844c09b8eefbf89815745
I20260928 02:04:25.301369 3449 ts_tablet_manager.cc:1939] T ea35626beb4844c09b8eefbf89815745 P 631f1ca6a4874be791cf70010b209203: Deleting consensus metadata
I20260928 02:04:25.302510 2864 catalog_manager.cc:4841] TS 631f1ca6a4874be791cf70010b209203 (127.1.182.131:46553): tablet ea35626beb4844c09b8eefbf89815745 (table test-table [id=1efe6c6f4ef04f1a9c0dd35b922aceb2]) successfully deleted
I20260928 02:04:25.304497 3448 ts_tablet_manager.cc:1918] T ea35626beb4844c09b8eefbf89815745 P 9f27378c91b242c4913a2035f885ce34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:04:25.304682 3448 log.cc:1196] T ea35626beb4844c09b8eefbf89815745 P 9f27378c91b242c4913a2035f885ce34: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/ea35626beb4844c09b8eefbf89815745
I20260928 02:04:25.305104 3450 ts_tablet_manager.cc:1918] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:04:25.305681 3448 ts_tablet_manager.cc:1939] T ea35626beb4844c09b8eefbf89815745 P 9f27378c91b242c4913a2035f885ce34: Deleting consensus metadata
I20260928 02:04:25.305857 3450 log.cc:1196] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561052730740-1754-0/minicluster-data/ts-0/wal/wals/ea35626beb4844c09b8eefbf89815745
I20260928 02:04:25.306381 3450 ts_tablet_manager.cc:1939] T ea35626beb4844c09b8eefbf89815745 P 95d757d0b10c450b8b6690168898b589: Deleting consensus metadata
I20260928 02:04:25.309485 2864 catalog_manager.cc:4841] TS 95d757d0b10c450b8b6690168898b589 (127.1.182.129:40771): tablet ea35626beb4844c09b8eefbf89815745 (table test-table [id=1efe6c6f4ef04f1a9c0dd35b922aceb2]) successfully deleted
I20260928 02:04:25.310052 2864 catalog_manager.cc:4841] TS 9f27378c91b242c4913a2035f885ce34 (127.1.182.130:33097): tablet ea35626beb4844c09b8eefbf89815745 (table test-table [id=1efe6c6f4ef04f1a9c0dd35b922aceb2]) successfully deleted
Sep 28 02:04:25 dist-test-slave-23bp krb5kdc[2834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561064, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:04:25.329098 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:25.288252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44610 (local address 127.1.182.129:40771)
0928 02:04:25.288748 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:25.288754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:25.311510 (+ 22756us) server_negotiation.cc:407] Connection header received
0928 02:04:25.311942 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:25.311955 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:25.312048 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:25.312123 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:25.314310 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:25.315611 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:25.317049 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:25.317535 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:25.317706 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:25.322246 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:25.322260 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:25.322276 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:25.322329 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:25.325915 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:25.326767 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:25.326782 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:25.326787 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:25.326960 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:25.327414 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:25.327420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:25.327425 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:25.327804 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:25.328156 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:25.328454 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 02:04:25.328717 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369}
W20260928 02:04:25.329891 3081 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44610
I20260928 02:04:25.351588 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:25.335547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57404 (local address 127.1.182.190:39221)
0928 02:04:25.337200 (+ 1653us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:25.337210 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:25.337725 (+ 515us) server_negotiation.cc:407] Connection header received
0928 02:04:25.337984 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:25.337991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:25.338071 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:25.338185 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:25.339807 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:25.340614 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:25.342070 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:25.342723 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:25.342805 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:25.345722 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:25.345729 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:25.345734 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:25.345776 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:25.348655 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:25.349337 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:25.349345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:25.349349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:25.349492 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:25.349932 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:25.349938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:25.349942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:25.350329 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:25.350734 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:25.350966 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 02:04:25.351227 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1435}
W20260928 02:04:25.352208 2877 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57404
I20260928 02:04:25.354760 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 2950
I20260928 02:04:25.368579 3429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:25.356270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44618 (local address 127.1.182.129:40771)
0928 02:04:25.356445 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:25.356450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:25.356468 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:04:25.356569 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:25.356581 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:25.356642 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:25.356748 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:25.357859 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:25.358685 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:25.359981 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:25.360426 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:25.360556 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:25.362656 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:25.362661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:25.362664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:25.362695 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:25.365964 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:25.366572 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:25.366578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:25.366581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:25.366713 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:25.367087 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:25.367092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:25.367096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:25.367388 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:25.367622 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:25.367977 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 02:04:25.368222 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 02:04:25.369112 3081 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44618
W20260928 02:04:25.369648 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 2950: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:04:25.369849 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 2950
I20260928 02:04:25.388634 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 3090
Sep 28 02:04:25 dist-test-slave-23bp krb5kdc[2834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561064, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.182.130@KRBTEST.COM
I20260928 02:04:25.403410 3456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:25.389380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57958 (local address 127.1.182.130:33097)
0928 02:04:25.389822 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:25.389826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:25.389842 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:04:25.389895 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:25.389898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:25.389962 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:25.390051 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:25.391222 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:25.391872 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:25.393023 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:25.393440 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:25.393527 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:25.397416 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:25.397429 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:25.397444 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:25.397475 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:25.400427 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:25.401170 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:25.401177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:25.401180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:25.401298 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:25.401724 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:25.401728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:25.401730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:25.402067 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:25.402367 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:25.402651 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 02:04:25.402874 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":250,"threads_started":1}
W20260928 02:04:25.404075 3220 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57958
W20260928 02:04:25.404685 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 3090: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:04:25.404863 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 3090
I20260928 02:04:25.421388 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 3229
Sep 28 02:04:25 dist-test-slave-23bp krb5kdc[2834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561064, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.182.131@KRBTEST.COM
I20260928 02:04:25.435523 3457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:25.422179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57086 (local address 127.1.182.131:46553)
0928 02:04:25.422681 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:25.422687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:25.422703 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:04:25.422755 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:25.422759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:25.422820 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:25.422908 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:25.424112 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:25.424810 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:25.425972 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:25.426387 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:25.426489 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:25.430555 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:25.430570 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:25.430586 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:25.430618 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:25.433206 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:25.433794 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:25.433801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:25.433803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:25.433924 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:25.434237 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:25.434240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:25.434242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:25.434559 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:25.434780 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:25.434999 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 02:04:25.435232 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":288,"threads_started":1}
W20260928 02:04:25.436224 3359 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57086
W20260928 02:04:25.436860 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 3229: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:04:25.436987 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 3229
I20260928 02:04:25.458530 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 2847
I20260928 02:04:25.459664 2919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:25.573550 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 2847
2026-09-28T02:04:25Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4725 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:04:25 dist-test-slave-23bp krb5kdc[3461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:04:25 dist-test-slave-23bp krb5kdc[3461](info): set up 2 sockets
Sep 28 02:04:25 dist-test-slave-23bp krb5kdc[3461](info): commencing operation
krb5kdc: starting...
W20260928 02:04:26.658140 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:04:26 dist-test-slave-23bp krb5kdc[3461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:04:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:04:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:04:26.846673 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:38331
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:38331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:27.170303 3474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:27.170636 3474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:27.170794 3474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:27.177903 3474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:27.178046 3474 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:04:27.178133 3474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:27.178247 3474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:27.178326 3474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:27.178414 3474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:27.188701 3474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:38331
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:38331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:27.190001 3474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:27.191375 3474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:27.203145 3479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:27.203203 3482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:27.203284 3480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:27.204152 3474 server_base.cc:1034] running on GCE node
I20260928 02:04:27.204943 3474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:27.206367 3474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:27.207597 3474 hybrid_clock.cc:648] HybridClock initialized: now 1790561067207539 us; error 42 us; skew 500 ppm
I20260928 02:04:27.207921 3474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:27 dist-test-slave-23bp krb5kdc[3461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561067, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:27.210975 3474 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:04:27.212508 3474 webserver.cc:460] Webserver started at http://127.1.182.190:36433/ using document root <none> and password file <none>
I20260928 02:04:27.213241 3474 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:27.213364 3474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:27.213675 3474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:27.216646 3474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "f72032c6bdad4992b9023be55432f25c"
format_stamp: "Formatted at 2026-09-28 02:04:27 on dist-test-slave-23bp"
server_key: "860d1f5ecc4e6212d5c152cbcb9bc294"
server_key_iv: "270055c6e0331f1e6b7d8c823b9f71d2"
server_key_version: "encryptionkey@0"
I20260928 02:04:27.217396 3474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "f72032c6bdad4992b9023be55432f25c"
format_stamp: "Formatted at 2026-09-28 02:04:27 on dist-test-slave-23bp"
server_key: "860d1f5ecc4e6212d5c152cbcb9bc294"
server_key_iv: "270055c6e0331f1e6b7d8c823b9f71d2"
server_key_version: "encryptionkey@0"
I20260928 02:04:27.222371 3474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 02:04:27.225838 3489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:27.227272 3474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:04:27.227451 3474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "f72032c6bdad4992b9023be55432f25c"
format_stamp: "Formatted at 2026-09-28 02:04:27 on dist-test-slave-23bp"
server_key: "860d1f5ecc4e6212d5c152cbcb9bc294"
server_key_iv: "270055c6e0331f1e6b7d8c823b9f71d2"
server_key_version: "encryptionkey@0"
I20260928 02:04:27.227658 3474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:27.248636 3474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:27.252511 3474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:27.252807 3474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:27.275615 3474 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:38331
I20260928 02:04:27.275640 3550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:38331 every 8 connection(s)
I20260928 02:04:27.277243 3474 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:04:27.281626 3551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:27.284758 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 3474
I20260928 02:04:27.284987 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:04:27.285591 1754 external_mini_cluster.cc:1428] Setting key ac273574e6644838ffeb78e1e1b1e8be
I20260928 02:04:27.294620 3551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c: Bootstrap starting.
I20260928 02:04:27.298213 3551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:27.299297 3551 log.cc:824] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:04:27 dist-test-slave-23bp krb5kdc[3461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:27.302138 3551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c: No bootstrap required, opened a new log
I20260928 02:04:27.309736 3554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:27.287554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56268 (local address 127.1.182.190:38331)
0928 02:04:27.288353 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:27.288368 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:27.288418 (+ 50us) server_negotiation.cc:407] Connection header received
0928 02:04:27.290393 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:27.290425 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:27.290939 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:27.291402 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:27.292589 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:27.294495 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:27.295827 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:27.296908 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:27.297079 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:27.301309 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:27.301319 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:27.301340 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:27.301391 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:27.305364 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:27.306014 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:27.306022 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:27.306030 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:27.306229 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:27.306559 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:27.306564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:27.306567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:27.307281 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:27.307569 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:27.307910 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 02:04:27.308354 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":311,"threads_started":1}
I20260928 02:04:27.312676 3551 raft_consensus.cc:348] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72032c6bdad4992b9023be55432f25c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38331 } }
I20260928 02:04:27.313030 3551 raft_consensus.cc:374] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:27.313103 3551 raft_consensus.cc:729] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f72032c6bdad4992b9023be55432f25c, State: Initialized, Role: FOLLOWER
I20260928 02:04:27.313568 3551 consensus_queue.cc:260] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72032c6bdad4992b9023be55432f25c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38331 } }
I20260928 02:04:27.313755 3551 raft_consensus.cc:388] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:04:27.313870 3551 raft_consensus.cc:482] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:04:27.314028 3551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:27.316386 3551 raft_consensus.cc:504] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72032c6bdad4992b9023be55432f25c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38331 } }
I20260928 02:04:27.316803 3551 leader_election.cc:302] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f72032c6bdad4992b9023be55432f25c; no voters:
I20260928 02:04:27.317312 3551 leader_election.cc:288] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:04:27.317451 3556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:27.318116 3556 raft_consensus.cc:686] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c [term 1 LEADER]: Becoming Leader. State: Replica: f72032c6bdad4992b9023be55432f25c, State: Running, Role: LEADER
I20260928 02:04:27.318712 3556 consensus_queue.cc:237] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72032c6bdad4992b9023be55432f25c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38331 } }
I20260928 02:04:27.319365 3551 sys_catalog.cc:564] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:27.321607 3557 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f72032c6bdad4992b9023be55432f25c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72032c6bdad4992b9023be55432f25c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38331 } } }
I20260928 02:04:27.322019 3557 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:27.322321 3558 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f72032c6bdad4992b9023be55432f25c. Latest consensus state: current_term: 1 leader_uuid: "f72032c6bdad4992b9023be55432f25c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72032c6bdad4992b9023be55432f25c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38331 } } }
I20260928 02:04:27.322670 3558 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:27.326813 3562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:27.332087 3562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:27.340292 3562 catalog_manager.cc:1347] Generated new cluster ID: e9a18e8000c141d39e93252a1d9a2a43
I20260928 02:04:27.340405 3562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:27.360657 3562 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:04:27.362540 3562 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:27.369550 3562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c: Generated new TSK 0
I20260928 02:04:27.370555 3562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.182.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
I20260928 02:04:27.422428 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:38331
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:27.753129 3577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:27.753454 3577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:27.753628 3577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:27.761359 3577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:27.761482 3577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:27.761638 3577 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:04:27.761718 3577 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:04:27.761824 3577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:04:27.771852 3577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.182.190:38331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:27.773188 3577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:27.774518 3577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:27.787868 3582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:27.787886 3583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:27.787945 3585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:27.788751 3577 server_base.cc:1034] running on GCE node
I20260928 02:04:27.789332 3577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:27.790308 3577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:27.791523 3577 hybrid_clock.cc:648] HybridClock initialized: now 1790561067791468 us; error 46 us; skew 500 ppm
I20260928 02:04:27.791888 3577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:27 dist-test-slave-23bp krb5kdc[3461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561067, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:27.794994 3577 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:04:27.796511 3577 webserver.cc:460] Webserver started at http://127.1.182.129:39217/ using document root <none> and password file <none>
I20260928 02:04:27.797165 3577 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:27.797268 3577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:27.797535 3577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:27.800633 3577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "ec96e0327eed44c3bf348feb2a47694a"
format_stamp: "Formatted at 2026-09-28 02:04:27 on dist-test-slave-23bp"
server_key: "a8d8496037c8477c49b966fbb14bad15"
server_key_iv: "ff23e333b6e6b9e4b242ad317beb1530"
server_key_version: "encryptionkey@0"
I20260928 02:04:27.801330 3577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "ec96e0327eed44c3bf348feb2a47694a"
format_stamp: "Formatted at 2026-09-28 02:04:27 on dist-test-slave-23bp"
server_key: "a8d8496037c8477c49b966fbb14bad15"
server_key_iv: "ff23e333b6e6b9e4b242ad317beb1530"
server_key_version: "encryptionkey@0"
I20260928 02:04:27.806401 3577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:04:27.809809 3592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:27.811398 3577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:04:27.811576 3577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "ec96e0327eed44c3bf348feb2a47694a"
format_stamp: "Formatted at 2026-09-28 02:04:27 on dist-test-slave-23bp"
server_key: "a8d8496037c8477c49b966fbb14bad15"
server_key_iv: "ff23e333b6e6b9e4b242ad317beb1530"
server_key_version: "encryptionkey@0"
I20260928 02:04:27.811779 3577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:27.824189 3577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:27.829007 3577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:27.829311 3577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:27.832129 3577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:27.832266 3577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:27.832391 3577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:27.832484 3577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:04:27 dist-test-slave-23bp krb5kdc[3461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561067, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:27.859131 3602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:27.833648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:38331 (local address 127.1.182.129:55077)
0928 02:04:27.834614 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:27.834661 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:27.836967 (+ 2306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:27.837894 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:27.837911 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:27.838690 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:27.839613 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:27.839632 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:27.841375 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:27.841382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:27.843525 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:27.843537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:27.844321 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:27.844331 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:27.844545 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:27.845302 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:27.845343 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:27.849066 (+ 3723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:27.853330 (+ 4264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:27.853345 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:27.853368 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:27.853959 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:27.854441 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:27.854448 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:27.854454 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:27.854664 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:27.855316 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:27.855329 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:27.855934 (+ 605us) client_negotiation.cc:769] Sending connection context
0928 02:04:27.856359 (+ 425us) client_negotiation.cc:240] Negotiation successful
0928 02:04:27.857054 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":399,"threads_started":1}
I20260928 02:04:27.859959 3601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:27.832993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:55077 (local address 127.1.182.190:38331)
0928 02:04:27.835472 (+ 2479us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:27.835479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:27.835527 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:04:27.837372 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:27.837382 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:27.837495 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:27.837638 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:27.839890 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:27.841210 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:27.843675 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:27.844083 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:27.844207 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:27.849317 (+ 5110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:27.849326 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:27.849332 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:27.849382 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:27.853161 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:27.854110 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:27.854119 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:27.854125 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:27.854300 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:27.854811 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:27.854816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:27.854819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:27.855147 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:27.855325 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:27.859283 (+ 3958us) server_negotiation.cc:299] Negotiation successful
0928 02:04:27.859541 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2325,"thread_start_us":2209,"threads_started":1}
I20260928 02:04:27.875290 3577 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:36093
I20260928 02:04:27.875325 3712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:36093 every 8 connection(s)
I20260928 02:04:27.876793 3577 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:04:27.878944 3713 heartbeater.cc:347] Connected to a master server at 127.1.182.190:38331
I20260928 02:04:27.879417 3713 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:27.880224 3713 heartbeater.cc:511] Master 127.1.182.190:38331 requested a full tablet report, sending...
I20260928 02:04:27.881714 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 3577
I20260928 02:04:27.881799 3601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:27.867680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:45347 (local address 127.1.182.190:38331)
0928 02:04:27.867911 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:27.867920 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:27.868452 (+ 532us) server_negotiation.cc:407] Connection header received
0928 02:04:27.868717 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:27.868724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:27.868815 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:27.868890 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:27.870171 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:27.871220 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:27.872502 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:27.873028 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:27.873174 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:27.875813 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:27.875818 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:27.875821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:27.875848 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:27.879038 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:27.879848 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:27.879853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:27.879856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:27.880009 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:27.880364 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:27.880368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:27.880371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:27.880711 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:27.880872 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:27.881244 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 02:04:27.881454 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:04:27.881897 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:04:27.881949 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:27.867644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:38331 (local address 127.1.182.129:45347)
0928 02:04:27.868325 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:27.868350 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:27.868529 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:27.869086 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:27.869089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:27.869582 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:27.870022 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:27.870036 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:27.871353 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:27.871363 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:27.872357 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:27.872367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:27.873245 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:27.873253 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:27.873440 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:27.874180 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:27.874208 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:27.875651 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:27.879198 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:27.879202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:27.879208 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:27.879738 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:27.880127 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:27.880130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:27.880133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:27.880241 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:27.880832 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:27.880835 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:27.880978 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 02:04:27.881128 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 02:04:27.881467 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":372,"threads_started":1}
I20260928 02:04:27.882562 3506 ts_manager.cc:194] Registered new tserver with Master: ec96e0327eed44c3bf348feb2a47694a (127.1.182.129:36093)
I20260928 02:04:27.882642 1754 external_mini_cluster.cc:1428] Setting key 82f2634a1de26d5663934cd19b61873f
I20260928 02:04:27.884496 3506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:55077
WARNING: no policy specified for kudu/127.1.182.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
I20260928 02:04:27.927098 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:38331
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:28.244961 3717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:28.245285 3717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:28.245456 3717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:28.252648 3717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:28.252791 3717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:28.252930 3717 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:04:28.253013 3717 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:04:28.253142 3717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:04:28.262917 3717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.182.190:38331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:28.264304 3717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:28.265552 3717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:28.278512 3723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:28.278504 3725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:28.279212 3722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:28.279848 3717 server_base.cc:1034] running on GCE node
I20260928 02:04:28.280445 3717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:28.281391 3717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:28.282663 3717 hybrid_clock.cc:648] HybridClock initialized: now 1790561068282604 us; error 46 us; skew 500 ppm
I20260928 02:04:28.282979 3717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:28 dist-test-slave-23bp krb5kdc[3461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561068, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:28.286305 3717 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:04:28.287976 3717 webserver.cc:460] Webserver started at http://127.1.182.130:42365/ using document root <none> and password file <none>
I20260928 02:04:28.288735 3717 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:28.288879 3717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:28.289203 3717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:28.292171 3717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "1ec164324ca148fcb9224f947a537cb5"
format_stamp: "Formatted at 2026-09-28 02:04:28 on dist-test-slave-23bp"
server_key: "31baf147a48f317513a2a6bd6e7aeb26"
server_key_iv: "116ab4651f11ef0da8482295a7c9af25"
server_key_version: "encryptionkey@0"
I20260928 02:04:28.292968 3717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "1ec164324ca148fcb9224f947a537cb5"
format_stamp: "Formatted at 2026-09-28 02:04:28 on dist-test-slave-23bp"
server_key: "31baf147a48f317513a2a6bd6e7aeb26"
server_key_iv: "116ab4651f11ef0da8482295a7c9af25"
server_key_version: "encryptionkey@0"
I20260928 02:04:28.298199 3717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:04:28.301738 3732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:28.303146 3717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:04:28.303328 3717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "1ec164324ca148fcb9224f947a537cb5"
format_stamp: "Formatted at 2026-09-28 02:04:28 on dist-test-slave-23bp"
server_key: "31baf147a48f317513a2a6bd6e7aeb26"
server_key_iv: "116ab4651f11ef0da8482295a7c9af25"
server_key_version: "encryptionkey@0"
I20260928 02:04:28.303535 3717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:28.336520 3717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:28.340562 3717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:28.340848 3717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:28.343595 3717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:28.343729 3717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:28.343861 3717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:28.343952 3717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:04:28 dist-test-slave-23bp krb5kdc[3461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561068, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:28.365814 3601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:28.344283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:49611 (local address 127.1.182.190:38331)
0928 02:04:28.344475 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:28.344481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:28.345980 (+ 1499us) server_negotiation.cc:407] Connection header received
0928 02:04:28.349027 (+ 3047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:28.349033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:28.349147 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:28.349276 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:28.351634 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:28.352498 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:28.354125 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:28.354568 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:28.354697 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:28.359457 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:28.359461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:28.359464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:28.359496 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:28.362756 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:28.363588 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:28.363593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:28.363597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:28.363721 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:28.364137 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:28.364142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:28.364145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:28.364427 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:28.364581 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:28.365252 (+ 671us) server_negotiation.cc:299] Negotiation successful
0928 02:04:28.365464 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:04:28.366959 3741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:28.344826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:38331 (local address 127.1.182.130:49611)
0928 02:04:28.345823 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:28.345869 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:28.348746 (+ 2877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:28.349718 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:28.349728 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:28.350512 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:28.351470 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:28.351490 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:28.352646 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:28.352651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:28.353990 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:28.354001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:28.354800 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:28.354808 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:28.355019 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:28.355817 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:28.355855 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:28.359174 (+ 3319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:28.362911 (+ 3737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:28.362920 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:28.362937 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:28.363469 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:28.363824 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:28.363830 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:28.363834 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:28.364027 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:28.364555 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:28.364563 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:28.364930 (+ 367us) client_negotiation.cc:769] Sending connection context
0928 02:04:28.365168 (+ 238us) client_negotiation.cc:240] Negotiation successful
0928 02:04:28.365677 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":519,"threads_started":1}
I20260928 02:04:28.394630 3806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:28.378626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:38331 (local address 127.1.182.130:36377)
0928 02:04:28.379298 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:28.379324 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:28.379576 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:28.380055 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:28.380059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:28.380515 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:28.380804 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:28.380813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:28.382613 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:28.382621 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:28.385167 (+ 2546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:28.385189 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:28.385970 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:28.385980 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:28.386253 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:28.386968 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:28.387000 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:28.388518 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:28.391588 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:28.391593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:28.391598 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:28.392315 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:28.392747 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:28.392752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:28.392756 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:28.392917 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:28.393710 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:28.393717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:28.393900 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 02:04:28.394003 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 02:04:28.394240 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":347,"threads_started":1}
I20260928 02:04:28.394996 3601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:28.378618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:36377 (local address 127.1.182.190:38331)
0928 02:04:28.378988 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:28.378997 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:28.379421 (+ 424us) server_negotiation.cc:407] Connection header received
0928 02:04:28.379752 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:28.379758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:28.379859 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:28.379996 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:28.380949 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:28.382420 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:28.385338 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:28.385759 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:28.385860 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:28.388747 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:28.388758 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:28.388761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:28.388792 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:28.391469 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:28.392470 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:28.392476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:28.392481 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:28.392623 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:28.393068 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:28.393073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:28.393076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:28.393572 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:28.393734 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:28.394183 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 02:04:28.394510 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260928 02:04:28.403393 3717 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:40627
I20260928 02:04:28.403425 3851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:40627 every 8 connection(s)
I20260928 02:04:28.404747 3717 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:04:28.406766 3852 heartbeater.cc:347] Connected to a master server at 127.1.182.190:38331
I20260928 02:04:28.407012 3852 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:28.407730 3852 heartbeater.cc:511] Master 127.1.182.190:38331 requested a full tablet report, sending...
I20260928 02:04:28.409191 3506 ts_manager.cc:194] Registered new tserver with Master: 1ec164324ca148fcb9224f947a537cb5 (127.1.182.130:40627)
I20260928 02:04:28.410074 3506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:49611
I20260928 02:04:28.414955 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 3717
I20260928 02:04:28.415132 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:04:28.415584 1754 external_mini_cluster.cc:1428] Setting key 1b90db6d8ea51b5f39888c974450c10c
WARNING: no policy specified for kudu/127.1.182.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
I20260928 02:04:28.457535 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:38331
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:28.776561 3856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:28.776862 3856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:28.777027 3856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:28.784441 3856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:28.784590 3856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:28.784725 3856 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:04:28.784811 3856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:04:28.784934 3856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:04:28.794934 3856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.182.190:38331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:28.796386 3856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:28.797736 3856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:28.810874 3861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:28.810935 3864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:28.810851 3862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:28.811893 3856 server_base.cc:1034] running on GCE node
I20260928 02:04:28.812500 3856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:28.813448 3856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:28.814673 3856 hybrid_clock.cc:648] HybridClock initialized: now 1790561068814589 us; error 53 us; skew 500 ppm
I20260928 02:04:28.814987 3856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:28 dist-test-slave-23bp krb5kdc[3461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561068, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:28.822237 3856 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:04:28.823804 3856 webserver.cc:460] Webserver started at http://127.1.182.131:34521/ using document root <none> and password file <none>
I20260928 02:04:28.824481 3856 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:28.824620 3856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:28.824930 3856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:28.827950 3856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "357aad85e8d64c6db2afcef44b4bb3b5"
format_stamp: "Formatted at 2026-09-28 02:04:28 on dist-test-slave-23bp"
server_key: "b03f1515650bb6688a36d95da8153e96"
server_key_iv: "a3b65b81810a30d15dd8501f1d2feac0"
server_key_version: "encryptionkey@0"
I20260928 02:04:28.828670 3856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "357aad85e8d64c6db2afcef44b4bb3b5"
format_stamp: "Formatted at 2026-09-28 02:04:28 on dist-test-slave-23bp"
server_key: "b03f1515650bb6688a36d95da8153e96"
server_key_iv: "a3b65b81810a30d15dd8501f1d2feac0"
server_key_version: "encryptionkey@0"
I20260928 02:04:28.834182 3856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:04:28.837456 3871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:28.838742 3856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:04:28.838901 3856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "357aad85e8d64c6db2afcef44b4bb3b5"
format_stamp: "Formatted at 2026-09-28 02:04:28 on dist-test-slave-23bp"
server_key: "b03f1515650bb6688a36d95da8153e96"
server_key_iv: "a3b65b81810a30d15dd8501f1d2feac0"
server_key_version: "encryptionkey@0"
I20260928 02:04:28.839120 3856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:28.853039 3856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:28.856760 3856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:28.857033 3856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:28.859576 3856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:28.859727 3856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:28.859867 3856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:28.859972 3856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:04:28 dist-test-slave-23bp krb5kdc[3461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561068, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:28.885000 3601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:28.860044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:55491 (local address 127.1.182.190:38331)
0928 02:04:28.860258 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:28.860267 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:28.861701 (+ 1434us) server_negotiation.cc:407] Connection header received
0928 02:04:28.864669 (+ 2968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:28.864675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:28.864754 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:28.864841 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:28.867414 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:28.868455 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:28.871534 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:28.872077 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:28.872225 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:28.877099 (+ 4874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:28.877109 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:28.877117 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:28.877172 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:28.881481 (+ 4309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:28.882291 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:28.882296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:28.882300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:28.882475 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:28.883204 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:28.883210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:28.883213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:28.883541 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:28.883731 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:28.884433 (+ 702us) server_negotiation.cc:299] Negotiation successful
0928 02:04:28.884660 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:04:28.886400 3880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:28.860627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:38331 (local address 127.1.182.131:55491)
0928 02:04:28.861521 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:28.861571 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:28.864348 (+ 2777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:28.865141 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:28.865155 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:28.866133 (+ 978us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:28.867189 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:28.867218 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:28.868626 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:28.868634 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:28.871363 (+ 2729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:28.871384 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:28.872340 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:28.872354 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:28.872749 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:28.873667 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:28.873707 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:28.876819 (+ 3112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:28.881640 (+ 4821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:28.881647 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:28.881659 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:28.882131 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:28.882638 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:28.882650 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:28.882657 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:28.882996 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:28.883681 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:28.883693 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:28.884037 (+ 344us) client_negotiation.cc:769] Sending connection context
0928 02:04:28.884308 (+ 271us) client_negotiation.cc:240] Negotiation successful
0928 02:04:28.884824 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":371,"threads_started":1}
I20260928 02:04:28.887581 3713 heartbeater.cc:503] Master 127.1.182.190:38331 was elected leader, sending a full tablet report...
I20260928 02:04:28.908414 3601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:28.893109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:36927 (local address 127.1.182.190:38331)
0928 02:04:28.893302 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:28.893310 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:28.894046 (+ 736us) server_negotiation.cc:407] Connection header received
0928 02:04:28.894193 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:28.894199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:28.894319 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:28.894437 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:28.895548 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:28.896980 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:28.898246 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:28.899201 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:28.899342 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:28.902106 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:28.902111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:28.902114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:28.902142 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:28.905630 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:28.906316 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:28.906323 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:28.906328 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:28.906477 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:28.906816 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:28.906822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:28.906826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:28.907225 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:28.907379 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:28.907809 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 02:04:28.908053 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:04:28.908553 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:28.893301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:38331 (local address 127.1.182.131:36927)
0928 02:04:28.893947 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:28.893971 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:28.894076 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:28.894486 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:28.894498 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:28.894896 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:28.895360 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:28.895381 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:28.897123 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:28.897126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:28.898107 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:28.898118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:28.899649 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:28.899658 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:28.899886 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:28.900737 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:28.900761 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:28.901863 (+ 1102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:28.905757 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:28.905765 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:28.905769 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:28.906199 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:28.906581 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:28.906587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:28.906591 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:28.906715 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:28.907345 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:28.907350 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:28.907571 (+ 221us) client_negotiation.cc:769] Sending connection context
0928 02:04:28.907717 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 02:04:28.908106 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":371,"threads_started":1}
I20260928 02:04:28.922389 3856 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:37609
I20260928 02:04:28.922421 3990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:37609 every 8 connection(s)
I20260928 02:04:28.923689 3856 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:04:28.925114 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 3856
I20260928 02:04:28.925294 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:04:28.925836 1754 external_mini_cluster.cc:1428] Setting key 9a153f3f4f219c42a01cf377823f14bc
I20260928 02:04:28.925704 3991 heartbeater.cc:347] Connected to a master server at 127.1.182.190:38331
I20260928 02:04:28.926689 3991 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:28.927467 3991 heartbeater.cc:511] Master 127.1.182.190:38331 requested a full tablet report, sending...
I20260928 02:04:28.928887 3506 ts_manager.cc:194] Registered new tserver with Master: 357aad85e8d64c6db2afcef44b4bb3b5 (127.1.182.131:37609)
I20260928 02:04:28.929818 3506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:55491
I20260928 02:04:28.940824 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:04:28 dist-test-slave-23bp krb5kdc[3461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:04:28 dist-test-slave-23bp krb5kdc[3461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:28.986650 3601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:28.972766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.1.182.190:38331)
0928 02:04:28.972986 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:28.972994 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:28.973209 (+ 215us) server_negotiation.cc:407] Connection header received
0928 02:04:28.973429 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:28.973433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:28.973493 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:28.973595 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:28.975139 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:28.975872 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:28.977218 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:28.977637 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:28.977771 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:28.981729 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:28.981735 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:28.981738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:28.981772 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:28.984129 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:28.984725 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:28.984732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:28.984735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:28.984857 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:28.985217 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:28.985223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:28.985226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:28.985484 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:28.985722 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:28.986050 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 02:04:28.986267 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:04:28.990576 3506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:04:28.997288 3506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:04:29.032553 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.008558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:40627 (local address 127.0.0.1:46204)
0928 02:04:29.020939 (+ 12381us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:29.021009 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:29.021269 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:29.024451 (+ 3182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:29.024458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:29.024487 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:29.024899 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.024914 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.026815 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.026821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.029387 (+ 2566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.029405 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.031531 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.031537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.031746 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.031802 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 02:04:29.031943 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 02:04:29.032144 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11976,"thread_start_us":15138,"threads_started":1}
I20260928 02:04:29.032755 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.009340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.1.182.129:36093)
0928 02:04:29.024200 (+ 14860us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:29.024208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:29.024231 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:04:29.024302 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:29.024306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:29.024545 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:29.024757 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:29.025615 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.027027 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.029884 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.031383 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.031500 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.031691 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:29.032310 (+ 619us) server_negotiation.cc:299] Negotiation successful
0928 02:04:29.032477 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14699,"thread_start_us":14593,"threads_started":1}
I20260928 02:04:29.033481 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.009055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:36093 (local address 127.0.0.1:59664)
0928 02:04:29.023473 (+ 14418us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:29.023495 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:29.023612 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:29.024960 (+ 1348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:29.024970 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:29.024997 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:29.025438 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.025452 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.027260 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.027266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.029725 (+ 2459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.029738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.031919 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.031927 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.032104 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.032134 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 02:04:29.033004 (+ 870us) client_negotiation.cc:240] Negotiation successful
0928 02:04:29.033137 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14291,"thread_start_us":12449,"threads_started":1}
I20260928 02:04:29.033936 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.008668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46204 (local address 127.1.182.130:40627)
0928 02:04:29.023769 (+ 15101us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:29.023778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:29.023806 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:04:29.023934 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:29.023945 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:29.024229 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:29.024436 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:29.025131 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.026649 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.029642 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.031063 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.031206 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.031411 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:29.033294 (+ 1883us) server_negotiation.cc:299] Negotiation successful
0928 02:04:29.033459 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14941,"thread_start_us":14830,"threads_started":1}
I20260928 02:04:29.037971 3643 tablet_service.cc:1467] Processing CreateTablet for tablet 1885124230294b30b15f0bd1dc5dd475 (DEFAULT_TABLE table=test-table [id=f2e04ea5838746cc8d8949f8703feda0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:29.038715 3785 tablet_service.cc:1467] Processing CreateTablet for tablet 1885124230294b30b15f0bd1dc5dd475 (DEFAULT_TABLE table=test-table [id=f2e04ea5838746cc8d8949f8703feda0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:29.039510 3643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1885124230294b30b15f0bd1dc5dd475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:29.040043 3785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1885124230294b30b15f0bd1dc5dd475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:29.040931 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.024800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:37609 (local address 127.0.0.1:34368)
0928 02:04:29.033029 (+ 8229us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:29.033047 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:29.033136 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:29.034844 (+ 1708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:29.034851 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:29.034877 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:29.035101 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.035116 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.036896 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.036902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.038053 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.038063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.040092 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.040181 (+ 89us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.040381 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.040414 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:04:29.040519 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:04:29.040651 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8127,"thread_start_us":10373,"threads_started":1}
I20260928 02:04:29.041036 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.024290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34368 (local address 127.1.182.131:37609)
0928 02:04:29.034177 (+ 9887us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:29.034187 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:29.034220 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:04:29.034320 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:29.034331 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:29.034579 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:29.034792 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:29.035319 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.036751 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.038207 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.039940 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.040083 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.040303 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:29.040568 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 02:04:29.040722 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9707,"thread_start_us":9525,"threads_started":1}
I20260928 02:04:29.043982 3925 tablet_service.cc:1467] Processing CreateTablet for tablet 1885124230294b30b15f0bd1dc5dd475 (DEFAULT_TABLE table=test-table [id=f2e04ea5838746cc8d8949f8703feda0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:29.045176 3925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1885124230294b30b15f0bd1dc5dd475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:29.053999 4011 tablet_bootstrap.cc:492] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5: Bootstrap starting.
I20260928 02:04:29.054745 4010 tablet_bootstrap.cc:492] T 1885124230294b30b15f0bd1dc5dd475 P ec96e0327eed44c3bf348feb2a47694a: Bootstrap starting.
I20260928 02:04:29.056360 4012 tablet_bootstrap.cc:492] T 1885124230294b30b15f0bd1dc5dd475 P 357aad85e8d64c6db2afcef44b4bb3b5: Bootstrap starting.
I20260928 02:04:29.058151 4010 tablet_bootstrap.cc:654] T 1885124230294b30b15f0bd1dc5dd475 P ec96e0327eed44c3bf348feb2a47694a: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:29.058662 4011 tablet_bootstrap.cc:654] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:29.059284 4010 log.cc:824] T 1885124230294b30b15f0bd1dc5dd475 P ec96e0327eed44c3bf348feb2a47694a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:29.059680 4012 tablet_bootstrap.cc:654] T 1885124230294b30b15f0bd1dc5dd475 P 357aad85e8d64c6db2afcef44b4bb3b5: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:29.059880 4011 log.cc:824] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:29.060990 4012 log.cc:824] T 1885124230294b30b15f0bd1dc5dd475 P 357aad85e8d64c6db2afcef44b4bb3b5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:29.062678 4010 tablet_bootstrap.cc:492] T 1885124230294b30b15f0bd1dc5dd475 P ec96e0327eed44c3bf348feb2a47694a: No bootstrap required, opened a new log
I20260928 02:04:29.062997 4010 ts_tablet_manager.cc:1397] T 1885124230294b30b15f0bd1dc5dd475 P ec96e0327eed44c3bf348feb2a47694a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260928 02:04:29.064425 4012 tablet_bootstrap.cc:492] T 1885124230294b30b15f0bd1dc5dd475 P 357aad85e8d64c6db2afcef44b4bb3b5: No bootstrap required, opened a new log
I20260928 02:04:29.064739 4012 ts_tablet_manager.cc:1397] T 1885124230294b30b15f0bd1dc5dd475 P 357aad85e8d64c6db2afcef44b4bb3b5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 02:04:29.069651 4011 tablet_bootstrap.cc:492] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5: No bootstrap required, opened a new log
I20260928 02:04:29.069921 4011 ts_tablet_manager.cc:1397] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260928 02:04:29.071264 4010 raft_consensus.cc:348] T 1885124230294b30b15f0bd1dc5dd475 P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } } peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } }
I20260928 02:04:29.071748 4010 raft_consensus.cc:374] T 1885124230294b30b15f0bd1dc5dd475 P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:29.071836 4010 raft_consensus.cc:729] T 1885124230294b30b15f0bd1dc5dd475 P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec96e0327eed44c3bf348feb2a47694a, State: Initialized, Role: FOLLOWER
I20260928 02:04:29.072459 4010 consensus_queue.cc:260] T 1885124230294b30b15f0bd1dc5dd475 P ec96e0327eed44c3bf348feb2a47694a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } } peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } }
I20260928 02:04:29.074119 4010 ts_tablet_manager.cc:1428] T 1885124230294b30b15f0bd1dc5dd475 P ec96e0327eed44c3bf348feb2a47694a: Time spent starting tablet: real 0.011s user 0.003s sys 0.007s
I20260928 02:04:29.076009 4012 raft_consensus.cc:348] T 1885124230294b30b15f0bd1dc5dd475 P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } } peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } }
I20260928 02:04:29.076354 4012 raft_consensus.cc:374] T 1885124230294b30b15f0bd1dc5dd475 P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:29.076468 4012 raft_consensus.cc:729] T 1885124230294b30b15f0bd1dc5dd475 P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357aad85e8d64c6db2afcef44b4bb3b5, State: Initialized, Role: FOLLOWER
I20260928 02:04:29.077019 4012 consensus_queue.cc:260] T 1885124230294b30b15f0bd1dc5dd475 P 357aad85e8d64c6db2afcef44b4bb3b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } } peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } }
I20260928 02:04:29.078118 3991 heartbeater.cc:503] Master 127.1.182.190:38331 was elected leader, sending a full tablet report...
I20260928 02:04:29.079700 4012 ts_tablet_manager.cc:1428] T 1885124230294b30b15f0bd1dc5dd475 P 357aad85e8d64c6db2afcef44b4bb3b5: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260928 02:04:29.079970 4011 raft_consensus.cc:348] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } } peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } }
I20260928 02:04:29.080375 4011 raft_consensus.cc:374] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:29.080485 4011 raft_consensus.cc:729] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec164324ca148fcb9224f947a537cb5, State: Initialized, Role: FOLLOWER
I20260928 02:04:29.081020 4011 consensus_queue.cc:260] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } } peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } }
I20260928 02:04:29.081952 3852 heartbeater.cc:503] Master 127.1.182.190:38331 was elected leader, sending a full tablet report...
I20260928 02:04:29.082581 4011 ts_tablet_manager.cc:1428] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20260928 02:04:29.129678 3714 tablet.cc:2369] T 1885124230294b30b15f0bd1dc5dd475 P ec96e0327eed44c3bf348feb2a47694a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:04:29.156809 3853 tablet.cc:2369] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:04:29.175220 3992 tablet.cc:2369] T 1885124230294b30b15f0bd1dc5dd475 P 357aad85e8d64c6db2afcef44b4bb3b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:04:29.275614 4018 raft_consensus.cc:482] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:29.275925 4018 raft_consensus.cc:504] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } } peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } }
I20260928 02:04:29.277555 4018 leader_election.cc:288] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 357aad85e8d64c6db2afcef44b4bb3b5 (127.1.182.131:37609), ec96e0327eed44c3bf348feb2a47694a (127.1.182.129:36093)
I20260928 02:04:29.284484 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.278249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:57857 (local address 127.1.182.131:37609)
0928 02:04:29.278467 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:29.278476 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:29.279099 (+ 623us) server_negotiation.cc:407] Connection header received
0928 02:04:29.279311 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:29.279315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:29.279372 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:29.279463 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:29.279951 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.280675 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.282566 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.283405 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.283593 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.283670 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:29.284113 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 02:04:29.284246 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:04:29.284613 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.277757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:37609 (local address 127.1.182.130:57857)
0928 02:04:29.278961 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:29.278981 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:29.279170 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:29.279477 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:29.279482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:29.279507 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:29.279823 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.279831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.280890 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.280899 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.282401 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.282414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.283626 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.283632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.283871 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.283913 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:04:29.284068 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 02:04:29.284219 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1076,"thread_start_us":6062,"threads_started":1}
I20260928 02:04:29.285637 3946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1885124230294b30b15f0bd1dc5dd475" candidate_uuid: "1ec164324ca148fcb9224f947a537cb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" is_pre_election: true
I20260928 02:04:29.286180 3946 raft_consensus.cc:2454] T 1885124230294b30b15f0bd1dc5dd475 P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ec164324ca148fcb9224f947a537cb5 in term 0.
I20260928 02:04:29.286834 3736 leader_election.cc:302] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ec164324ca148fcb9224f947a537cb5, 357aad85e8d64c6db2afcef44b4bb3b5; no voters:
I20260928 02:04:29.287268 4018 raft_consensus.cc:2781] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:04:29.287452 4018 raft_consensus.cc:482] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:04:29.287653 4018 raft_consensus.cc:3037] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:29.290675 4018 raft_consensus.cc:504] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } } peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } }
I20260928 02:04:29.291415 4018 leader_election.cc:288] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 [CANDIDATE]: Term 1 election: Requested vote from peers 357aad85e8d64c6db2afcef44b4bb3b5 (127.1.182.131:37609), ec96e0327eed44c3bf348feb2a47694a (127.1.182.129:36093)
I20260928 02:04:29.291790 3946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1885124230294b30b15f0bd1dc5dd475" candidate_uuid: "1ec164324ca148fcb9224f947a537cb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "357aad85e8d64c6db2afcef44b4bb3b5"
I20260928 02:04:29.292001 3946 raft_consensus.cc:3037] T 1885124230294b30b15f0bd1dc5dd475 P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:29.292120 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.278563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:49811 (local address 127.1.182.129:36093)
0928 02:04:29.278747 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:29.278752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:29.284258 (+ 5506us) server_negotiation.cc:407] Connection header received
0928 02:04:29.284527 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:29.284534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:29.284630 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:29.284699 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:29.285655 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.286473 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.289713 (+ 3240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.290953 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.291128 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.291197 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:29.291754 (+ 557us) server_negotiation.cc:299] Negotiation successful
0928 02:04:29.291895 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:04:29.292435 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.277978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:36093 (local address 127.1.182.130:49811)
0928 02:04:29.284163 (+ 6185us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:29.284182 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:29.284367 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:29.285227 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:29.285230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:29.285251 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:29.285507 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.285514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.287187 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.287193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.289527 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.289545 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.291182 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.291188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.291554 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.291614 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 02:04:29.291717 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 02:04:29.291856 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6041,"thread_start_us":782,"threads_started":1}
I20260928 02:04:29.293530 3662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1885124230294b30b15f0bd1dc5dd475" candidate_uuid: "1ec164324ca148fcb9224f947a537cb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec96e0327eed44c3bf348feb2a47694a"
I20260928 02:04:29.293553 3663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1885124230294b30b15f0bd1dc5dd475" candidate_uuid: "1ec164324ca148fcb9224f947a537cb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec96e0327eed44c3bf348feb2a47694a" is_pre_election: true
I20260928 02:04:29.293998 3662 raft_consensus.cc:3037] T 1885124230294b30b15f0bd1dc5dd475 P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:29.294993 3946 raft_consensus.cc:2454] T 1885124230294b30b15f0bd1dc5dd475 P 357aad85e8d64c6db2afcef44b4bb3b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ec164324ca148fcb9224f947a537cb5 in term 1.
I20260928 02:04:29.295394 3736 leader_election.cc:302] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ec164324ca148fcb9224f947a537cb5, 357aad85e8d64c6db2afcef44b4bb3b5; no voters:
I20260928 02:04:29.295691 4018 raft_consensus.cc:2781] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:29.296921 4018 raft_consensus.cc:686] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 [term 1 LEADER]: Becoming Leader. State: Replica: 1ec164324ca148fcb9224f947a537cb5, State: Running, Role: LEADER
I20260928 02:04:29.297245 3662 raft_consensus.cc:2454] T 1885124230294b30b15f0bd1dc5dd475 P ec96e0327eed44c3bf348feb2a47694a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ec164324ca148fcb9224f947a537cb5 in term 1.
I20260928 02:04:29.297464 4018 consensus_queue.cc:237] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } } peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } }
I20260928 02:04:29.302018 3505 catalog_manager.cc:5495] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ec164324ca148fcb9224f947a537cb5 (127.1.182.130). New cstate: current_term: 1 leader_uuid: "1ec164324ca148fcb9224f947a537cb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:04:29 dist-test-slave-23bp krb5kdc[3461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561067, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:29.359099 3601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.345417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43816 (local address 127.1.182.190:38331)
0928 02:04:29.345590 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:29.345595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:29.346089 (+ 494us) server_negotiation.cc:407] Connection header received
0928 02:04:29.346227 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:29.346231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:29.346308 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:29.346390 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:29.347480 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.348208 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.349805 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.350503 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.350628 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.354147 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:29.354153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:29.354165 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:29.354200 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:29.356468 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:29.357106 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:29.357111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:29.357115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:29.357233 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:29.357575 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:29.357580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:29.357583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:29.357841 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:29.357981 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:29.358372 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 02:04:29.358709 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":70}
I20260928 02:04:29.359109 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.345487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:38331 (local address 127.0.0.1:43816)
0928 02:04:29.345998 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:29.346014 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:29.346133 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:29.346432 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:29.346435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:29.347106 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:29.347358 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.347367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.348315 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.348318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.349641 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.349658 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.350693 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.350705 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.350897 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.351672 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:29.351698 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:29.353986 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:29.356592 (+ 2606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:29.356605 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:29.356612 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:29.356987 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:29.357329 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:29.357336 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:29.357339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:29.357472 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:29.357966 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:29.357974 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:29.358132 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 02:04:29.358291 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 02:04:29.358577 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":288,"threads_started":1}
I20260928 02:04:29.363927 3505 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:43816:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:04:29.364857 3505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:04:29.371202 3925 tablet_service.cc:1467] Processing CreateTablet for tablet 42c2daed37a544948ea3189b7b16d0aa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=33d4ceb987644b4abc58fa27cfd78f23]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:04:29.371035 3785 tablet_service.cc:1467] Processing CreateTablet for tablet 42c2daed37a544948ea3189b7b16d0aa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=33d4ceb987644b4abc58fa27cfd78f23]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:04:29.370937 3643 tablet_service.cc:1467] Processing CreateTablet for tablet 42c2daed37a544948ea3189b7b16d0aa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=33d4ceb987644b4abc58fa27cfd78f23]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:04:29.371887 3925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42c2daed37a544948ea3189b7b16d0aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:29.371929 3643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42c2daed37a544948ea3189b7b16d0aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:29.371929 3785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42c2daed37a544948ea3189b7b16d0aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:29.376946 4012 tablet_bootstrap.cc:492] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5: Bootstrap starting.
I20260928 02:04:29.378316 4012 tablet_bootstrap.cc:654] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:29.378849 4010 tablet_bootstrap.cc:492] T 42c2daed37a544948ea3189b7b16d0aa P ec96e0327eed44c3bf348feb2a47694a: Bootstrap starting.
I20260928 02:04:29.379554 4011 tablet_bootstrap.cc:492] T 42c2daed37a544948ea3189b7b16d0aa P 1ec164324ca148fcb9224f947a537cb5: Bootstrap starting.
I20260928 02:04:29.380445 4010 tablet_bootstrap.cc:654] T 42c2daed37a544948ea3189b7b16d0aa P ec96e0327eed44c3bf348feb2a47694a: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:29.381222 4011 tablet_bootstrap.cc:654] T 42c2daed37a544948ea3189b7b16d0aa P 1ec164324ca148fcb9224f947a537cb5: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:29.382568 4010 tablet_bootstrap.cc:492] T 42c2daed37a544948ea3189b7b16d0aa P ec96e0327eed44c3bf348feb2a47694a: No bootstrap required, opened a new log
I20260928 02:04:29.382678 4012 tablet_bootstrap.cc:492] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5: No bootstrap required, opened a new log
I20260928 02:04:29.382725 4010 ts_tablet_manager.cc:1397] T 42c2daed37a544948ea3189b7b16d0aa P ec96e0327eed44c3bf348feb2a47694a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 02:04:29.382797 4012 ts_tablet_manager.cc:1397] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260928 02:04:29.383512 4012 raft_consensus.cc:348] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:29.383513 4010 raft_consensus.cc:348] T 42c2daed37a544948ea3189b7b16d0aa P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:29.383770 4012 raft_consensus.cc:374] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:29.383770 4010 raft_consensus.cc:374] T 42c2daed37a544948ea3189b7b16d0aa P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:29.383869 4012 raft_consensus.cc:729] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357aad85e8d64c6db2afcef44b4bb3b5, State: Initialized, Role: FOLLOWER
I20260928 02:04:29.383869 4010 raft_consensus.cc:729] T 42c2daed37a544948ea3189b7b16d0aa P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec96e0327eed44c3bf348feb2a47694a, State: Initialized, Role: FOLLOWER
I20260928 02:04:29.384100 4010 consensus_queue.cc:260] T 42c2daed37a544948ea3189b7b16d0aa P ec96e0327eed44c3bf348feb2a47694a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:29.384093 4012 consensus_queue.cc:260] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:29.384250 4011 tablet_bootstrap.cc:492] T 42c2daed37a544948ea3189b7b16d0aa P 1ec164324ca148fcb9224f947a537cb5: No bootstrap required, opened a new log
I20260928 02:04:29.384531 4017 tablet_replica.cc:439] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } } }
I20260928 02:04:29.384552 4016 tablet_replica.cc:439] T 42c2daed37a544948ea3189b7b16d0aa P ec96e0327eed44c3bf348feb2a47694a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } } }
I20260928 02:04:29.384895 4017 tablet_replica.cc:442] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:04:29.384910 4016 tablet_replica.cc:442] T 42c2daed37a544948ea3189b7b16d0aa P ec96e0327eed44c3bf348feb2a47694a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:04:29.385365 4010 ts_tablet_manager.cc:1428] T 42c2daed37a544948ea3189b7b16d0aa P ec96e0327eed44c3bf348feb2a47694a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:04:29.384385 4011 ts_tablet_manager.cc:1397] T 42c2daed37a544948ea3189b7b16d0aa P 1ec164324ca148fcb9224f947a537cb5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260928 02:04:29.386178 4012 ts_tablet_manager.cc:1428] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:04:29.386238 4011 raft_consensus.cc:348] T 42c2daed37a544948ea3189b7b16d0aa P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:29.386497 4011 raft_consensus.cc:374] T 42c2daed37a544948ea3189b7b16d0aa P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:29.386588 4011 raft_consensus.cc:729] T 42c2daed37a544948ea3189b7b16d0aa P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec164324ca148fcb9224f947a537cb5, State: Initialized, Role: FOLLOWER
I20260928 02:04:29.386821 4011 consensus_queue.cc:260] T 42c2daed37a544948ea3189b7b16d0aa P 1ec164324ca148fcb9224f947a537cb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:29.387385 4011 ts_tablet_manager.cc:1428] T 42c2daed37a544948ea3189b7b16d0aa P 1ec164324ca148fcb9224f947a537cb5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 02:04:29.387266 4018 tablet_replica.cc:439] T 42c2daed37a544948ea3189b7b16d0aa P 1ec164324ca148fcb9224f947a537cb5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } } }
I20260928 02:04:29.387585 4018 tablet_replica.cc:442] T 42c2daed37a544948ea3189b7b16d0aa P 1ec164324ca148fcb9224f947a537cb5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:04:29.560017 4017 raft_consensus.cc:482] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:29.560285 4017 raft_consensus.cc:504] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:29.561829 4017 leader_election.cc:288] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec96e0327eed44c3bf348feb2a47694a (127.1.182.129:36093), 1ec164324ca148fcb9224f947a537cb5 (127.1.182.130:40627)
I20260928 02:04:29.569540 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.562274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:58267 (local address 127.1.182.129:36093)
0928 02:04:29.562512 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:29.562521 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:29.563555 (+ 1034us) server_negotiation.cc:407] Connection header received
0928 02:04:29.563769 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:29.563775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:29.563851 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:29.563958 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:29.564473 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.565393 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.567612 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.568462 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.568651 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.568743 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:29.569035 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 02:04:29.569171 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 02:04:29.569648 4034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.562407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:55699 (local address 127.1.182.130:40627)
0928 02:04:29.563146 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:29.563152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:29.563642 (+ 490us) server_negotiation.cc:407] Connection header received
0928 02:04:29.563802 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:29.563805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:29.563895 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:29.564012 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:29.564707 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.565498 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.567615 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.568612 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.568790 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.568861 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:29.569162 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 02:04:29.569284 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":390,"threads_started":1}
I20260928 02:04:29.570086 4033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.562195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:40627 (local address 127.1.182.131:55699)
0928 02:04:29.563533 (+ 1338us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:29.563551 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:29.563679 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:29.564038 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:29.564042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:29.564113 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:29.564520 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.564530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.565639 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.565644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.567462 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.567472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.568822 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.568828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.569020 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.569042 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:04:29.569139 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 02:04:29.569274 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1246,"mutex_wait_us":44,"thread_start_us":1121,"threads_started":1}
I20260928 02:04:29.570061 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.562049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:36093 (local address 127.1.182.131:58267)
0928 02:04:29.563433 (+ 1384us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:29.563454 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:29.563619 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:29.563975 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:29.563980 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:29.564017 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:29.564279 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.564288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.565541 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.565550 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.567458 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.567470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.568652 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.568658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.568860 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.568886 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:04:29.569011 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 02:04:29.569154 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1244,"mutex_wait_us":17,"spinlock_wait_cycles":26368,"thread_start_us":1029,"threads_started":1}
I20260928 02:04:29.570789 3662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c2daed37a544948ea3189b7b16d0aa" candidate_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec96e0327eed44c3bf348feb2a47694a" is_pre_election: true
I20260928 02:04:29.571143 3662 raft_consensus.cc:2454] T 42c2daed37a544948ea3189b7b16d0aa P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 357aad85e8d64c6db2afcef44b4bb3b5 in term 0.
I20260928 02:04:29.571231 3807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c2daed37a544948ea3189b7b16d0aa" candidate_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ec164324ca148fcb9224f947a537cb5" is_pre_election: true
I20260928 02:04:29.571672 3807 raft_consensus.cc:2454] T 42c2daed37a544948ea3189b7b16d0aa P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 357aad85e8d64c6db2afcef44b4bb3b5 in term 0.
I20260928 02:04:29.571631 3873 leader_election.cc:302] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357aad85e8d64c6db2afcef44b4bb3b5, ec96e0327eed44c3bf348feb2a47694a; no voters:
I20260928 02:04:29.571985 4017 raft_consensus.cc:2781] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:04:29.572118 4017 raft_consensus.cc:482] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:04:29.572218 4017 raft_consensus.cc:3037] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:29.574257 4017 raft_consensus.cc:504] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:29.574805 4017 leader_election.cc:288] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5 [CANDIDATE]: Term 1 election: Requested vote from peers ec96e0327eed44c3bf348feb2a47694a (127.1.182.129:36093), 1ec164324ca148fcb9224f947a537cb5 (127.1.182.130:40627)
I20260928 02:04:29.575161 3662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c2daed37a544948ea3189b7b16d0aa" candidate_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec96e0327eed44c3bf348feb2a47694a"
I20260928 02:04:29.575284 3807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c2daed37a544948ea3189b7b16d0aa" candidate_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ec164324ca148fcb9224f947a537cb5"
I20260928 02:04:29.575397 3662 raft_consensus.cc:3037] T 42c2daed37a544948ea3189b7b16d0aa P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:29.575511 3807 raft_consensus.cc:3037] T 42c2daed37a544948ea3189b7b16d0aa P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:29.578686 3662 raft_consensus.cc:2454] T 42c2daed37a544948ea3189b7b16d0aa P ec96e0327eed44c3bf348feb2a47694a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357aad85e8d64c6db2afcef44b4bb3b5 in term 1.
I20260928 02:04:29.578703 3807 raft_consensus.cc:2454] T 42c2daed37a544948ea3189b7b16d0aa P 1ec164324ca148fcb9224f947a537cb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357aad85e8d64c6db2afcef44b4bb3b5 in term 1.
I20260928 02:04:29.579106 3873 leader_election.cc:302] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357aad85e8d64c6db2afcef44b4bb3b5, ec96e0327eed44c3bf348feb2a47694a; no voters:
I20260928 02:04:29.579388 4017 raft_consensus.cc:2781] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:29.579787 4017 raft_consensus.cc:686] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5 [term 1 LEADER]: Becoming Leader. State: Replica: 357aad85e8d64c6db2afcef44b4bb3b5, State: Running, Role: LEADER
I20260928 02:04:29.580291 4017 consensus_queue.cc:237] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:29.583004 4035 tablet_replica.cc:439] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5: TxnStatusTablet state changed. Reason: New leader 357aad85e8d64c6db2afcef44b4bb3b5. Latest consensus state: current_term: 1 leader_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } } }
I20260928 02:04:29.583410 4035 tablet_replica.cc:442] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:04:29.584131 4037 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:04:29.584393 4037 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:04:29.584868 3506 catalog_manager.cc:5495] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 357aad85e8d64c6db2afcef44b4bb3b5 (127.1.182.131). New cstate: current_term: 1 leader_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } health_report { overall_health: HEALTHY } } }
I20260928 02:04:29.627591 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.621253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:37609 (local address 127.0.0.1:34380)
0928 02:04:29.621487 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:29.621506 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:29.621639 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:29.622087 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:29.622090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:29.622118 (+ 28us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:04:29.622298 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.622305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.623865 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.623874 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.624902 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.624913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.625888 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.625893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.626040 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.626091 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:04:29.627162 (+ 1071us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:04:29.627191 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 02:04:29.627257 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:04:29.627361 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 02:04:29.627681 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.621466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34380 (local address 127.1.182.131:37609)
0928 02:04:29.621653 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:29.621660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:29.621681 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:04:29.621805 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:29.621810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:29.621910 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:29.622084 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:04:29.622427 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.623741 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.625026 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.625736 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.625854 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.626265 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:04:29.627029 (+ 764us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:04:29.627206 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:29.627291 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 02:04:29.627370 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:04:29.635114 3662 raft_consensus.cc:1261] T 42c2daed37a544948ea3189b7b16d0aa P ec96e0327eed44c3bf348feb2a47694a [term 1 FOLLOWER]: Refusing update from remote peer 357aad85e8d64c6db2afcef44b4bb3b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:04:29.635155 3807 raft_consensus.cc:1261] T 42c2daed37a544948ea3189b7b16d0aa P 1ec164324ca148fcb9224f947a537cb5 [term 1 FOLLOWER]: Refusing update from remote peer 357aad85e8d64c6db2afcef44b4bb3b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:04:29.635974 4035 consensus_queue.cc:1035] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:29.636286 4017 consensus_queue.cc:1035] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:29.643765 4016 tablet_replica.cc:439] T 42c2daed37a544948ea3189b7b16d0aa P ec96e0327eed44c3bf348feb2a47694a: TxnStatusTablet state changed. Reason: New leader 357aad85e8d64c6db2afcef44b4bb3b5. Latest consensus state: current_term: 1 leader_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } } }
I20260928 02:04:29.644004 4016 tablet_replica.cc:442] T 42c2daed37a544948ea3189b7b16d0aa P ec96e0327eed44c3bf348feb2a47694a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:04:29.648870 4040 mvcc.cc:205] Tried to move back new op lower bound from 7334138141216485376 to 7334138141004615680. Current Snapshot: MvccSnapshot[applied={T|T < 7334138141216485376}]
I20260928 02:04:29.652541 4017 tablet_replica.cc:439] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } } }
I20260928 02:04:29.652902 4017 tablet_replica.cc:442] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:04:29.656527 4018 tablet_replica.cc:439] T 42c2daed37a544948ea3189b7b16d0aa P 1ec164324ca148fcb9224f947a537cb5: TxnStatusTablet state changed. Reason: New leader 357aad85e8d64c6db2afcef44b4bb3b5. Latest consensus state: current_term: 1 leader_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } } }
I20260928 02:04:29.656857 4018 tablet_replica.cc:442] T 42c2daed37a544948ea3189b7b16d0aa P 1ec164324ca148fcb9224f947a537cb5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:04:29.657104 4017 tablet_replica.cc:439] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } } }
I20260928 02:04:29.657373 4017 tablet_replica.cc:442] T 42c2daed37a544948ea3189b7b16d0aa P 357aad85e8d64c6db2afcef44b4bb3b5: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:04:29.658871 4042 mvcc.cc:205] Tried to move back new op lower bound from 7334138141216485376 to 7334138141004615680. Current Snapshot: MvccSnapshot[applied={T|T < 7334138141216485376}]
I20260928 02:04:29.664577 4016 tablet_replica.cc:439] T 42c2daed37a544948ea3189b7b16d0aa P ec96e0327eed44c3bf348feb2a47694a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } } }
I20260928 02:04:29.664848 4016 tablet_replica.cc:442] T 42c2daed37a544948ea3189b7b16d0aa P ec96e0327eed44c3bf348feb2a47694a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:04:29.667016 4018 tablet_replica.cc:439] T 42c2daed37a544948ea3189b7b16d0aa P 1ec164324ca148fcb9224f947a537cb5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } } }
I20260928 02:04:29.667330 4018 tablet_replica.cc:442] T 42c2daed37a544948ea3189b7b16d0aa P 1ec164324ca148fcb9224f947a537cb5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:04:29.674613 4034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.659969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46208 (local address 127.1.182.130:40627)
0928 02:04:29.660125 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:29.660131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:29.669223 (+ 9092us) server_negotiation.cc:407] Connection header received
0928 02:04:29.669526 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:29.669555 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:29.669645 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:29.669828 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:04:29.670296 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.671177 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.672550 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.672953 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.673081 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.673473 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:04:29.673982 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:04:29.674148 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:29.674198 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 02:04:29.674275 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:04:29.690874 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.684927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:37609 (local address 127.1.182.130:44875)
0928 02:04:29.685480 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:29.685499 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:29.685608 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:29.685943 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:29.685946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:29.685955 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:04:29.686209 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.686216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.687600 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.687603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.688715 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.688725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.689832 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.689841 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.689967 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.690018 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:04:29.690419 (+ 401us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:04:29.690426 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:04:29.690486 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 02:04:29.690576 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":284,"threads_started":1}
I20260928 02:04:29.690904 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.685044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:44875 (local address 127.1.182.131:37609)
0928 02:04:29.685248 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:29.685254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:29.685574 (+ 320us) server_negotiation.cc:407] Connection header received
0928 02:04:29.685731 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:29.685737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:29.685826 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:29.685917 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:04:29.686341 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.687475 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.688883 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.689609 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.689777 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.690204 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:04:29.690332 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:04:29.690421 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:29.690527 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 02:04:29.690610 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:04:29.693014 4018 consensus_queue.cc:1035] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:29.701944 4021 consensus_queue.cc:1035] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:29.736670 4058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.729406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:40627 (local address 127.1.182.131:35827)
0928 02:04:29.730004 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:29.730040 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:29.730207 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:29.731115 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:29.731120 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:29.731136 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:04:29.731524 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.731533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.733137 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.733141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.734511 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:29.734525 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.735636 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.735640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:29.735768 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.735821 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:04:29.736199 (+ 378us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:04:29.736206 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:04:29.736262 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:04:29.736354 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":289,"threads_started":1}
I20260928 02:04:29.736656 4034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:29.729503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:35827 (local address 127.1.182.130:40627)
0928 02:04:29.729659 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:29.729666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:29.730123 (+ 457us) server_negotiation.cc:407] Connection header received
0928 02:04:29.730846 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:29.730852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:29.730964 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:29.731061 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:04:29.731745 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.733012 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.734644 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:29.735449 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:29.735586 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:29.735979 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:04:29.736103 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:04:29.736201 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:29.736281 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 02:04:29.736356 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:04:29.773499 3506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:04:29.773788 3506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:04:29.777765 3506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c: Sending DeleteTablet for 3 replicas of tablet 1885124230294b30b15f0bd1dc5dd475
I20260928 02:04:29.779186 3643 tablet_service.cc:1514] Processing DeleteTablet for tablet 1885124230294b30b15f0bd1dc5dd475 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:29 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:59664
I20260928 02:04:29.780452 3785 tablet_service.cc:1514] Processing DeleteTablet for tablet 1885124230294b30b15f0bd1dc5dd475 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:29 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:46204
I20260928 02:04:29.781597 3925 tablet_service.cc:1514] Processing DeleteTablet for tablet 1885124230294b30b15f0bd1dc5dd475 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:29 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:34368
I20260928 02:04:29.785535 3506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:04:29.787774 3506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:04:29.800074 3920 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3498ea1f5240988edd81e0bc5b3e03 (DEFAULT_TABLE table=test-table [id=45b1052fcb4b43c0a19353b66fd258a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:29.800709 3920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3498ea1f5240988edd81e0bc5b3e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:29.800710 3784 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3498ea1f5240988edd81e0bc5b3e03 (DEFAULT_TABLE table=test-table [id=45b1052fcb4b43c0a19353b66fd258a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:29.801258 3784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3498ea1f5240988edd81e0bc5b3e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:29.803647 3642 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3498ea1f5240988edd81e0bc5b3e03 (DEFAULT_TABLE table=test-table [id=45b1052fcb4b43c0a19353b66fd258a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:29.804196 3642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3498ea1f5240988edd81e0bc5b3e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:29.808117 4011 tablet_bootstrap.cc:492] T 6b3498ea1f5240988edd81e0bc5b3e03 P 1ec164324ca148fcb9224f947a537cb5: Bootstrap starting.
I20260928 02:04:29.809947 4011 tablet_bootstrap.cc:654] T 6b3498ea1f5240988edd81e0bc5b3e03 P 1ec164324ca148fcb9224f947a537cb5: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:29.815755 4063 tablet_replica.cc:330] T 1885124230294b30b15f0bd1dc5dd475 P 357aad85e8d64c6db2afcef44b4bb3b5: stopping tablet replica
I20260928 02:04:29.816620 4010 tablet_bootstrap.cc:492] T 6b3498ea1f5240988edd81e0bc5b3e03 P ec96e0327eed44c3bf348feb2a47694a: Bootstrap starting.
I20260928 02:04:29.818074 4063 raft_consensus.cc:2229] T 1885124230294b30b15f0bd1dc5dd475 P 357aad85e8d64c6db2afcef44b4bb3b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:04:29.816655 4062 tablet_replica.cc:330] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5: stopping tablet replica
I20260928 02:04:29.818339 4010 tablet_bootstrap.cc:654] T 6b3498ea1f5240988edd81e0bc5b3e03 P ec96e0327eed44c3bf348feb2a47694a: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:29.819095 4063 raft_consensus.cc:2258] T 1885124230294b30b15f0bd1dc5dd475 P 357aad85e8d64c6db2afcef44b4bb3b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:29.819247 4062 raft_consensus.cc:2229] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:04:29.820631 4062 raft_consensus.cc:2258] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:29.821626 4061 tablet_replica.cc:330] T 1885124230294b30b15f0bd1dc5dd475 P ec96e0327eed44c3bf348feb2a47694a: stopping tablet replica
I20260928 02:04:29.822602 4012 tablet_bootstrap.cc:492] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5: Bootstrap starting.
I20260928 02:04:29.823031 4062 ts_tablet_manager.cc:1905] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:29.823370 4061 raft_consensus.cc:2229] T 1885124230294b30b15f0bd1dc5dd475 P ec96e0327eed44c3bf348feb2a47694a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:04:29.823760 4011 tablet_bootstrap.cc:492] T 6b3498ea1f5240988edd81e0bc5b3e03 P 1ec164324ca148fcb9224f947a537cb5: No bootstrap required, opened a new log
I20260928 02:04:29.823773 4061 raft_consensus.cc:2258] T 1885124230294b30b15f0bd1dc5dd475 P ec96e0327eed44c3bf348feb2a47694a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:29.823944 4011 ts_tablet_manager.cc:1397] T 6b3498ea1f5240988edd81e0bc5b3e03 P 1ec164324ca148fcb9224f947a537cb5: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.010s
I20260928 02:04:29.824291 4012 tablet_bootstrap.cc:654] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:29.825151 4011 raft_consensus.cc:348] T 6b3498ea1f5240988edd81e0bc5b3e03 P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:29.825443 4011 raft_consensus.cc:374] T 6b3498ea1f5240988edd81e0bc5b3e03 P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:29.825533 4011 raft_consensus.cc:729] T 6b3498ea1f5240988edd81e0bc5b3e03 P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec164324ca148fcb9224f947a537cb5, State: Initialized, Role: FOLLOWER
I20260928 02:04:29.825819 4011 consensus_queue.cc:260] T 6b3498ea1f5240988edd81e0bc5b3e03 P 1ec164324ca148fcb9224f947a537cb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:29.826216 4061 ts_tablet_manager.cc:1905] T 1885124230294b30b15f0bd1dc5dd475 P ec96e0327eed44c3bf348feb2a47694a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:29.827229 4011 ts_tablet_manager.cc:1428] T 6b3498ea1f5240988edd81e0bc5b3e03 P 1ec164324ca148fcb9224f947a537cb5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:04:29.830137 4062 ts_tablet_manager.cc:1918] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:04:29.830315 4062 log.cc:1196] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/1885124230294b30b15f0bd1dc5dd475
I20260928 02:04:29.830892 4062 ts_tablet_manager.cc:1939] T 1885124230294b30b15f0bd1dc5dd475 P 1ec164324ca148fcb9224f947a537cb5: Deleting consensus metadata
I20260928 02:04:29.839979 4061 ts_tablet_manager.cc:1918] T 1885124230294b30b15f0bd1dc5dd475 P ec96e0327eed44c3bf348feb2a47694a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:04:29.840204 4061 log.cc:1196] T 1885124230294b30b15f0bd1dc5dd475 P ec96e0327eed44c3bf348feb2a47694a: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-0/wal/wals/1885124230294b30b15f0bd1dc5dd475
I20260928 02:04:29.840588 4063 ts_tablet_manager.cc:1905] T 1885124230294b30b15f0bd1dc5dd475 P 357aad85e8d64c6db2afcef44b4bb3b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:29.840728 4061 ts_tablet_manager.cc:1939] T 1885124230294b30b15f0bd1dc5dd475 P ec96e0327eed44c3bf348feb2a47694a: Deleting consensus metadata
I20260928 02:04:29.841064 3493 catalog_manager.cc:4841] TS 1ec164324ca148fcb9224f947a537cb5 (127.1.182.130:40627): tablet 1885124230294b30b15f0bd1dc5dd475 (table test-table [id=f2e04ea5838746cc8d8949f8703feda0]) successfully deleted
I20260928 02:04:29.842367 3491 catalog_manager.cc:4841] TS ec96e0327eed44c3bf348feb2a47694a (127.1.182.129:36093): tablet 1885124230294b30b15f0bd1dc5dd475 (table test-table [id=f2e04ea5838746cc8d8949f8703feda0]) successfully deleted
I20260928 02:04:29.845729 4012 tablet_bootstrap.cc:492] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5: No bootstrap required, opened a new log
I20260928 02:04:29.845916 4012 ts_tablet_manager.cc:1397] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.009s
I20260928 02:04:29.846792 4012 raft_consensus.cc:348] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:29.847424 4012 raft_consensus.cc:374] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:29.847539 4012 raft_consensus.cc:729] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357aad85e8d64c6db2afcef44b4bb3b5, State: Initialized, Role: FOLLOWER
I20260928 02:04:29.847868 4012 consensus_queue.cc:260] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:29.849857 4063 ts_tablet_manager.cc:1918] T 1885124230294b30b15f0bd1dc5dd475 P 357aad85e8d64c6db2afcef44b4bb3b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:04:29.850028 4063 log.cc:1196] T 1885124230294b30b15f0bd1dc5dd475 P 357aad85e8d64c6db2afcef44b4bb3b5: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/1885124230294b30b15f0bd1dc5dd475
I20260928 02:04:29.850589 4063 ts_tablet_manager.cc:1939] T 1885124230294b30b15f0bd1dc5dd475 P 357aad85e8d64c6db2afcef44b4bb3b5: Deleting consensus metadata
I20260928 02:04:29.851046 4012 ts_tablet_manager.cc:1428] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260928 02:04:29.853219 3493 catalog_manager.cc:4841] TS 357aad85e8d64c6db2afcef44b4bb3b5 (127.1.182.131:37609): tablet 1885124230294b30b15f0bd1dc5dd475 (table test-table [id=f2e04ea5838746cc8d8949f8703feda0]) successfully deleted
I20260928 02:04:29.854666 4010 tablet_bootstrap.cc:492] T 6b3498ea1f5240988edd81e0bc5b3e03 P ec96e0327eed44c3bf348feb2a47694a: No bootstrap required, opened a new log
I20260928 02:04:29.854843 4010 ts_tablet_manager.cc:1397] T 6b3498ea1f5240988edd81e0bc5b3e03 P ec96e0327eed44c3bf348feb2a47694a: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.003s
I20260928 02:04:29.855625 4010 raft_consensus.cc:348] T 6b3498ea1f5240988edd81e0bc5b3e03 P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:29.855861 4010 raft_consensus.cc:374] T 6b3498ea1f5240988edd81e0bc5b3e03 P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:29.855930 4010 raft_consensus.cc:729] T 6b3498ea1f5240988edd81e0bc5b3e03 P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec96e0327eed44c3bf348feb2a47694a, State: Initialized, Role: FOLLOWER
I20260928 02:04:29.856241 4010 consensus_queue.cc:260] T 6b3498ea1f5240988edd81e0bc5b3e03 P ec96e0327eed44c3bf348feb2a47694a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:29.856793 4010 ts_tablet_manager.cc:1428] T 6b3498ea1f5240988edd81e0bc5b3e03 P ec96e0327eed44c3bf348feb2a47694a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:04:29.957614 4017 raft_consensus.cc:482] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:29.957859 4017 raft_consensus.cc:504] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:29.958596 4017 leader_election.cc:288] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec96e0327eed44c3bf348feb2a47694a (127.1.182.129:36093), 1ec164324ca148fcb9224f947a537cb5 (127.1.182.130:40627)
I20260928 02:04:29.959581 3807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3498ea1f5240988edd81e0bc5b3e03" candidate_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ec164324ca148fcb9224f947a537cb5" is_pre_election: true
I20260928 02:04:29.959868 3807 raft_consensus.cc:2454] T 6b3498ea1f5240988edd81e0bc5b3e03 P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 357aad85e8d64c6db2afcef44b4bb3b5 in term 0.
I20260928 02:04:29.960903 3662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3498ea1f5240988edd81e0bc5b3e03" candidate_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec96e0327eed44c3bf348feb2a47694a" is_pre_election: true
I20260928 02:04:29.961169 3662 raft_consensus.cc:2454] T 6b3498ea1f5240988edd81e0bc5b3e03 P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 357aad85e8d64c6db2afcef44b4bb3b5 in term 0.
I20260928 02:04:29.961778 3875 leader_election.cc:302] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ec164324ca148fcb9224f947a537cb5, 357aad85e8d64c6db2afcef44b4bb3b5; no voters:
I20260928 02:04:29.962855 4017 raft_consensus.cc:2781] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:04:29.962981 4017 raft_consensus.cc:482] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:04:29.963474 4017 raft_consensus.cc:3037] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:29.967095 4017 raft_consensus.cc:504] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:29.968305 3662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3498ea1f5240988edd81e0bc5b3e03" candidate_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec96e0327eed44c3bf348feb2a47694a"
I20260928 02:04:29.968536 3662 raft_consensus.cc:3037] T 6b3498ea1f5240988edd81e0bc5b3e03 P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:29.971608 3662 raft_consensus.cc:2454] T 6b3498ea1f5240988edd81e0bc5b3e03 P ec96e0327eed44c3bf348feb2a47694a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357aad85e8d64c6db2afcef44b4bb3b5 in term 1.
I20260928 02:04:29.972072 3807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3498ea1f5240988edd81e0bc5b3e03" candidate_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ec164324ca148fcb9224f947a537cb5"
I20260928 02:04:29.972169 3873 leader_election.cc:302] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357aad85e8d64c6db2afcef44b4bb3b5, ec96e0327eed44c3bf348feb2a47694a; no voters:
I20260928 02:04:29.972293 3807 raft_consensus.cc:3037] T 6b3498ea1f5240988edd81e0bc5b3e03 P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:29.972540 4053 raft_consensus.cc:2781] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:29.972735 4053 raft_consensus.cc:686] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 [term 1 LEADER]: Becoming Leader. State: Replica: 357aad85e8d64c6db2afcef44b4bb3b5, State: Running, Role: LEADER
I20260928 02:04:29.973055 4053 consensus_queue.cc:237] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:29.975848 3807 raft_consensus.cc:2454] T 6b3498ea1f5240988edd81e0bc5b3e03 P 1ec164324ca148fcb9224f947a537cb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357aad85e8d64c6db2afcef44b4bb3b5 in term 1.
I20260928 02:04:29.976424 4017 leader_election.cc:288] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 [CANDIDATE]: Term 1 election: Requested vote from peers ec96e0327eed44c3bf348feb2a47694a (127.1.182.129:36093), 1ec164324ca148fcb9224f947a537cb5 (127.1.182.130:40627)
I20260928 02:04:29.978785 3506 catalog_manager.cc:5495] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 357aad85e8d64c6db2afcef44b4bb3b5 (127.1.182.131). New cstate: current_term: 1 leader_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } health_report { overall_health: HEALTHY } } }
I20260928 02:04:30.025709 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:30.018986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34394 (local address 127.1.182.131:37609)
0928 02:04:30.019224 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:30.019232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:30.019489 (+ 257us) server_negotiation.cc:407] Connection header received
0928 02:04:30.019714 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:30.019724 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:30.019832 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:30.019956 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:04:30.020342 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:30.021583 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:30.023575 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:30.024272 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:30.024399 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:30.024903 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:04:30.025049 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:04:30.025191 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:30.025297 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 02:04:30.025377 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:04:30.050374 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:30.045557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:40353 (local address 127.1.182.131:37609)
0928 02:04:30.045726 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:30.045732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:30.045876 (+ 144us) server_negotiation.cc:407] Connection header received
0928 02:04:30.046065 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:30.046070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:30.046158 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:30.046243 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:04:30.046611 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:30.047431 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:30.048710 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:30.049131 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:30.049266 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:30.049599 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:04:30.049713 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:04:30.049802 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:30.049908 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 02:04:30.049994 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":48}
I20260928 02:04:30.050374 4058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:30.045547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:37609 (local address 127.1.182.131:40353)
0928 02:04:30.045788 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:30.045805 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:30.045946 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:30.046252 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:30.046257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:30.046267 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:04:30.046501 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:30.046509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:30.047555 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:30.047563 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:30.048575 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:30.048586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:30.049271 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:30.049276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:30.049422 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:30.049452 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:04:30.049799 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:04:30.049810 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 02:04:30.049894 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 02:04:30.050028 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":40}
I20260928 02:04:30.059854 3807 raft_consensus.cc:1261] T 6b3498ea1f5240988edd81e0bc5b3e03 P 1ec164324ca148fcb9224f947a537cb5 [term 1 FOLLOWER]: Refusing update from remote peer 357aad85e8d64c6db2afcef44b4bb3b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:04:30.060325 3662 raft_consensus.cc:1261] T 6b3498ea1f5240988edd81e0bc5b3e03 P ec96e0327eed44c3bf348feb2a47694a [term 1 FOLLOWER]: Refusing update from remote peer 357aad85e8d64c6db2afcef44b4bb3b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:04:30.061002 4017 consensus_queue.cc:1035] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:30.061313 4053 consensus_queue.cc:1035] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:30.074976 4041 mvcc.cc:205] Tried to move back new op lower bound from 7334138142960873472 to 7334138142613004288. Current Snapshot: MvccSnapshot[applied={T|T < 7334138142960873472}]
W20260928 02:04:30.105618 4040 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:04:31.703971 3506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:04:31.704201 3506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:04:31.707980 3506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c: Sending DeleteTablet for 3 replicas of tablet 6b3498ea1f5240988edd81e0bc5b3e03
I20260928 02:04:31.708865 3642 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b3498ea1f5240988edd81e0bc5b3e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:31 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:59664
I20260928 02:04:31.708845 3785 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b3498ea1f5240988edd81e0bc5b3e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:31 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:46204
I20260928 02:04:31.708966 3920 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b3498ea1f5240988edd81e0bc5b3e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:31 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:34368
I20260928 02:04:31.719907 4093 tablet_replica.cc:330] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5: stopping tablet replica
I20260928 02:04:31.720124 4093 raft_consensus.cc:2229] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:04:31.720625 4093 raft_consensus.cc:2258] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:31.721199 4091 tablet_replica.cc:330] T 6b3498ea1f5240988edd81e0bc5b3e03 P ec96e0327eed44c3bf348feb2a47694a: stopping tablet replica
I20260928 02:04:31.721395 4091 raft_consensus.cc:2229] T 6b3498ea1f5240988edd81e0bc5b3e03 P ec96e0327eed44c3bf348feb2a47694a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:04:31.721616 4091 raft_consensus.cc:2258] T 6b3498ea1f5240988edd81e0bc5b3e03 P ec96e0327eed44c3bf348feb2a47694a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:31.722343 4093 ts_tablet_manager.cc:1905] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:31.723702 4091 ts_tablet_manager.cc:1905] T 6b3498ea1f5240988edd81e0bc5b3e03 P ec96e0327eed44c3bf348feb2a47694a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:31.726531 4092 tablet_replica.cc:330] T 6b3498ea1f5240988edd81e0bc5b3e03 P 1ec164324ca148fcb9224f947a537cb5: stopping tablet replica
I20260928 02:04:31.726723 4092 raft_consensus.cc:2229] T 6b3498ea1f5240988edd81e0bc5b3e03 P 1ec164324ca148fcb9224f947a537cb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:04:31.726962 4092 raft_consensus.cc:2258] T 6b3498ea1f5240988edd81e0bc5b3e03 P 1ec164324ca148fcb9224f947a537cb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:31.728704 4093 ts_tablet_manager.cc:1918] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:04:31.728833 4093 log.cc:1196] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/6b3498ea1f5240988edd81e0bc5b3e03
I20260928 02:04:31.729269 4093 ts_tablet_manager.cc:1939] T 6b3498ea1f5240988edd81e0bc5b3e03 P 357aad85e8d64c6db2afcef44b4bb3b5: Deleting consensus metadata
I20260928 02:04:31.730459 3493 catalog_manager.cc:4841] TS 357aad85e8d64c6db2afcef44b4bb3b5 (127.1.182.131:37609): tablet 6b3498ea1f5240988edd81e0bc5b3e03 (table test-table [id=45b1052fcb4b43c0a19353b66fd258a3]) successfully deleted
I20260928 02:04:31.731135 4092 ts_tablet_manager.cc:1905] T 6b3498ea1f5240988edd81e0bc5b3e03 P 1ec164324ca148fcb9224f947a537cb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:31.731465 4091 ts_tablet_manager.cc:1918] T 6b3498ea1f5240988edd81e0bc5b3e03 P ec96e0327eed44c3bf348feb2a47694a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:04:31.731674 4091 log.cc:1196] T 6b3498ea1f5240988edd81e0bc5b3e03 P ec96e0327eed44c3bf348feb2a47694a: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-0/wal/wals/6b3498ea1f5240988edd81e0bc5b3e03
I20260928 02:04:31.732190 4091 ts_tablet_manager.cc:1939] T 6b3498ea1f5240988edd81e0bc5b3e03 P ec96e0327eed44c3bf348feb2a47694a: Deleting consensus metadata
I20260928 02:04:31.733217 3491 catalog_manager.cc:4841] TS ec96e0327eed44c3bf348feb2a47694a (127.1.182.129:36093): tablet 6b3498ea1f5240988edd81e0bc5b3e03 (table test-table [id=45b1052fcb4b43c0a19353b66fd258a3]) successfully deleted
I20260928 02:04:31.737007 4092 ts_tablet_manager.cc:1918] T 6b3498ea1f5240988edd81e0bc5b3e03 P 1ec164324ca148fcb9224f947a537cb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:04:31.737143 4092 log.cc:1196] T 6b3498ea1f5240988edd81e0bc5b3e03 P 1ec164324ca148fcb9224f947a537cb5: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/6b3498ea1f5240988edd81e0bc5b3e03
I20260928 02:04:31.737550 4092 ts_tablet_manager.cc:1939] T 6b3498ea1f5240988edd81e0bc5b3e03 P 1ec164324ca148fcb9224f947a537cb5: Deleting consensus metadata
I20260928 02:04:31.738415 3493 catalog_manager.cc:4841] TS 1ec164324ca148fcb9224f947a537cb5 (127.1.182.130:40627): tablet 6b3498ea1f5240988edd81e0bc5b3e03 (table test-table [id=45b1052fcb4b43c0a19353b66fd258a3]) successfully deleted
Sep 28 02:04:31 dist-test-slave-23bp krb5kdc[3461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561071, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:04:31 dist-test-slave-23bp krb5kdc[3461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561071, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:31.765269 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:31.751565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43828 (local address 127.1.182.190:38331)
0928 02:04:31.752156 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:31.752163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:31.752286 (+ 123us) server_negotiation.cc:407] Connection header received
0928 02:04:31.752495 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:31.752499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:31.752563 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:31.752674 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:31.753933 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:31.754706 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:31.756031 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:31.756416 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:31.756547 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:31.760285 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:31.760299 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:31.760303 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:31.760336 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:31.763034 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:31.763766 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:31.763770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:31.763772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:31.763906 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:31.764197 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:31.764200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:31.764202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:31.764441 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:31.764602 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:31.764862 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 02:04:31.765011 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":275,"threads_started":1}
I20260928 02:04:31.768486 3506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:04:31.769090 3506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:04:31.774993 3642 tablet_service.cc:1467] Processing CreateTablet for tablet 57b79cfdab654183ab7446bbc6c2e31f (DEFAULT_TABLE table=test-table [id=bff6d7bfeadb427788ea0e311664b7f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:31.775210 3785 tablet_service.cc:1467] Processing CreateTablet for tablet 57b79cfdab654183ab7446bbc6c2e31f (DEFAULT_TABLE table=test-table [id=bff6d7bfeadb427788ea0e311664b7f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:31.775494 3642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b79cfdab654183ab7446bbc6c2e31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:31.775617 3785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b79cfdab654183ab7446bbc6c2e31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:31.775583 3920 tablet_service.cc:1467] Processing CreateTablet for tablet 57b79cfdab654183ab7446bbc6c2e31f (DEFAULT_TABLE table=test-table [id=bff6d7bfeadb427788ea0e311664b7f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:31.776018 3920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b79cfdab654183ab7446bbc6c2e31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:31.781575 4102 tablet_bootstrap.cc:492] T 57b79cfdab654183ab7446bbc6c2e31f P ec96e0327eed44c3bf348feb2a47694a: Bootstrap starting.
I20260928 02:04:31.782641 4104 tablet_bootstrap.cc:492] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5: Bootstrap starting.
I20260928 02:04:31.783294 4102 tablet_bootstrap.cc:654] T 57b79cfdab654183ab7446bbc6c2e31f P ec96e0327eed44c3bf348feb2a47694a: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:31.783414 4103 tablet_bootstrap.cc:492] T 57b79cfdab654183ab7446bbc6c2e31f P 1ec164324ca148fcb9224f947a537cb5: Bootstrap starting.
I20260928 02:04:31.784888 4104 tablet_bootstrap.cc:654] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:31.785171 4103 tablet_bootstrap.cc:654] T 57b79cfdab654183ab7446bbc6c2e31f P 1ec164324ca148fcb9224f947a537cb5: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:31.786203 4102 tablet_bootstrap.cc:492] T 57b79cfdab654183ab7446bbc6c2e31f P ec96e0327eed44c3bf348feb2a47694a: No bootstrap required, opened a new log
I20260928 02:04:31.786348 4102 ts_tablet_manager.cc:1397] T 57b79cfdab654183ab7446bbc6c2e31f P ec96e0327eed44c3bf348feb2a47694a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 02:04:31.787031 4104 tablet_bootstrap.cc:492] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5: No bootstrap required, opened a new log
I20260928 02:04:31.787202 4104 ts_tablet_manager.cc:1397] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:04:31.787160 4102 raft_consensus.cc:348] T 57b79cfdab654183ab7446bbc6c2e31f P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:31.787421 4102 raft_consensus.cc:374] T 57b79cfdab654183ab7446bbc6c2e31f P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:31.787549 4102 raft_consensus.cc:729] T 57b79cfdab654183ab7446bbc6c2e31f P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec96e0327eed44c3bf348feb2a47694a, State: Initialized, Role: FOLLOWER
I20260928 02:04:31.787921 4103 tablet_bootstrap.cc:492] T 57b79cfdab654183ab7446bbc6c2e31f P 1ec164324ca148fcb9224f947a537cb5: No bootstrap required, opened a new log
I20260928 02:04:31.787814 4102 consensus_queue.cc:260] T 57b79cfdab654183ab7446bbc6c2e31f P ec96e0327eed44c3bf348feb2a47694a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:31.787954 4104 raft_consensus.cc:348] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:31.788174 4103 ts_tablet_manager.cc:1397] T 57b79cfdab654183ab7446bbc6c2e31f P 1ec164324ca148fcb9224f947a537cb5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:04:31.788196 4104 raft_consensus.cc:374] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:31.788309 4104 raft_consensus.cc:729] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357aad85e8d64c6db2afcef44b4bb3b5, State: Initialized, Role: FOLLOWER
I20260928 02:04:31.788594 4104 consensus_queue.cc:260] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:31.789356 4104 ts_tablet_manager.cc:1428] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:04:31.789620 4103 raft_consensus.cc:348] T 57b79cfdab654183ab7446bbc6c2e31f P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:31.789872 4103 raft_consensus.cc:374] T 57b79cfdab654183ab7446bbc6c2e31f P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:31.789983 4103 raft_consensus.cc:729] T 57b79cfdab654183ab7446bbc6c2e31f P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec164324ca148fcb9224f947a537cb5, State: Initialized, Role: FOLLOWER
I20260928 02:04:31.790259 4103 consensus_queue.cc:260] T 57b79cfdab654183ab7446bbc6c2e31f P 1ec164324ca148fcb9224f947a537cb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:31.791567 4102 ts_tablet_manager.cc:1428] T 57b79cfdab654183ab7446bbc6c2e31f P ec96e0327eed44c3bf348feb2a47694a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:04:31.792410 4103 ts_tablet_manager.cc:1428] T 57b79cfdab654183ab7446bbc6c2e31f P 1ec164324ca148fcb9224f947a537cb5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 02:04:31.816906 4074 raft_consensus.cc:482] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:31.817095 4074 raft_consensus.cc:504] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:31.817895 4074 leader_election.cc:288] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec96e0327eed44c3bf348feb2a47694a (127.1.182.129:36093), 1ec164324ca148fcb9224f947a537cb5 (127.1.182.130:40627)
I20260928 02:04:31.818356 3663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b79cfdab654183ab7446bbc6c2e31f" candidate_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec96e0327eed44c3bf348feb2a47694a" is_pre_election: true
I20260928 02:04:31.818436 3807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b79cfdab654183ab7446bbc6c2e31f" candidate_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ec164324ca148fcb9224f947a537cb5" is_pre_election: true
I20260928 02:04:31.818595 3663 raft_consensus.cc:2454] T 57b79cfdab654183ab7446bbc6c2e31f P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 357aad85e8d64c6db2afcef44b4bb3b5 in term 0.
I20260928 02:04:31.818717 3807 raft_consensus.cc:2454] T 57b79cfdab654183ab7446bbc6c2e31f P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 357aad85e8d64c6db2afcef44b4bb3b5 in term 0.
I20260928 02:04:31.819118 3873 leader_election.cc:302] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357aad85e8d64c6db2afcef44b4bb3b5, ec96e0327eed44c3bf348feb2a47694a; no voters:
I20260928 02:04:31.819396 4074 raft_consensus.cc:2781] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:04:31.819491 4074 raft_consensus.cc:482] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:04:31.819592 4074 raft_consensus.cc:3037] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:31.821631 4074 raft_consensus.cc:504] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:31.822141 4074 leader_election.cc:288] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 [CANDIDATE]: Term 1 election: Requested vote from peers ec96e0327eed44c3bf348feb2a47694a (127.1.182.129:36093), 1ec164324ca148fcb9224f947a537cb5 (127.1.182.130:40627)
I20260928 02:04:31.822682 3663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b79cfdab654183ab7446bbc6c2e31f" candidate_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec96e0327eed44c3bf348feb2a47694a"
I20260928 02:04:31.822713 3807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b79cfdab654183ab7446bbc6c2e31f" candidate_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ec164324ca148fcb9224f947a537cb5"
I20260928 02:04:31.822890 3807 raft_consensus.cc:3037] T 57b79cfdab654183ab7446bbc6c2e31f P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:31.822908 3663 raft_consensus.cc:3037] T 57b79cfdab654183ab7446bbc6c2e31f P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:31.826756 3663 raft_consensus.cc:2454] T 57b79cfdab654183ab7446bbc6c2e31f P ec96e0327eed44c3bf348feb2a47694a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357aad85e8d64c6db2afcef44b4bb3b5 in term 1.
I20260928 02:04:31.826819 3807 raft_consensus.cc:2454] T 57b79cfdab654183ab7446bbc6c2e31f P 1ec164324ca148fcb9224f947a537cb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357aad85e8d64c6db2afcef44b4bb3b5 in term 1.
I20260928 02:04:31.827327 3875 leader_election.cc:302] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ec164324ca148fcb9224f947a537cb5, 357aad85e8d64c6db2afcef44b4bb3b5; no voters:
I20260928 02:04:31.827708 4074 raft_consensus.cc:2781] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:31.827912 4074 raft_consensus.cc:686] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 [term 1 LEADER]: Becoming Leader. State: Replica: 357aad85e8d64c6db2afcef44b4bb3b5, State: Running, Role: LEADER
I20260928 02:04:31.828260 4074 consensus_queue.cc:237] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:31.831590 3505 catalog_manager.cc:5495] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 357aad85e8d64c6db2afcef44b4bb3b5 (127.1.182.131). New cstate: current_term: 1 leader_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } health_report { overall_health: HEALTHY } } }
I20260928 02:04:31.861131 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:31.854025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34406 (local address 127.1.182.131:37609)
0928 02:04:31.854973 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:31.854977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:31.855001 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:04:31.855049 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:31.855055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:31.855178 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:31.855291 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:04:31.855889 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:31.857026 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:31.859111 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:31.859695 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:31.859832 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:31.860274 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:04:31.860404 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:04:31.860504 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:31.860648 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 02:04:31.860723 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":649,"threads_started":1}
I20260928 02:04:31.871369 3807 raft_consensus.cc:1261] T 57b79cfdab654183ab7446bbc6c2e31f P 1ec164324ca148fcb9224f947a537cb5 [term 1 FOLLOWER]: Refusing update from remote peer 357aad85e8d64c6db2afcef44b4bb3b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:04:31.871390 3663 raft_consensus.cc:1261] T 57b79cfdab654183ab7446bbc6c2e31f P ec96e0327eed44c3bf348feb2a47694a [term 1 FOLLOWER]: Refusing update from remote peer 357aad85e8d64c6db2afcef44b4bb3b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:04:31.871990 4074 consensus_queue.cc:1035] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:31.872287 4071 consensus_queue.cc:1035] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:31.918025 3504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:04:31.918239 3504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:04:31.921211 3504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c: Sending DeleteTablet for 3 replicas of tablet 57b79cfdab654183ab7446bbc6c2e31f
I20260928 02:04:31.921942 3642 tablet_service.cc:1514] Processing DeleteTablet for tablet 57b79cfdab654183ab7446bbc6c2e31f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:31 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:59664
I20260928 02:04:31.922127 3920 tablet_service.cc:1514] Processing DeleteTablet for tablet 57b79cfdab654183ab7446bbc6c2e31f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:31 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:34368
I20260928 02:04:31.922150 3785 tablet_service.cc:1514] Processing DeleteTablet for tablet 57b79cfdab654183ab7446bbc6c2e31f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:31 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:46204
I20260928 02:04:31.922397 4092 tablet_replica.cc:330] T 57b79cfdab654183ab7446bbc6c2e31f P 1ec164324ca148fcb9224f947a537cb5: stopping tablet replica
I20260928 02:04:31.922602 4092 raft_consensus.cc:2229] T 57b79cfdab654183ab7446bbc6c2e31f P 1ec164324ca148fcb9224f947a537cb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:04:31.922793 4092 raft_consensus.cc:2258] T 57b79cfdab654183ab7446bbc6c2e31f P 1ec164324ca148fcb9224f947a537cb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:31.923137 4093 tablet_replica.cc:330] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5: stopping tablet replica
I20260928 02:04:31.923357 4093 raft_consensus.cc:2229] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:04:31.923682 4093 raft_consensus.cc:2258] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:31.926846 4091 tablet_replica.cc:330] T 57b79cfdab654183ab7446bbc6c2e31f P ec96e0327eed44c3bf348feb2a47694a: stopping tablet replica
I20260928 02:04:31.927101 4091 raft_consensus.cc:2229] T 57b79cfdab654183ab7446bbc6c2e31f P ec96e0327eed44c3bf348feb2a47694a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:04:31.927302 4091 raft_consensus.cc:2258] T 57b79cfdab654183ab7446bbc6c2e31f P ec96e0327eed44c3bf348feb2a47694a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:31.928418 4093 ts_tablet_manager.cc:1905] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:31.932734 3504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:04:31.932983 4092 ts_tablet_manager.cc:1905] T 57b79cfdab654183ab7446bbc6c2e31f P 1ec164324ca148fcb9224f947a537cb5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 02:04:31.933463 3504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:04:31.935349 4091 ts_tablet_manager.cc:1905] T 57b79cfdab654183ab7446bbc6c2e31f P ec96e0327eed44c3bf348feb2a47694a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:31.940646 3785 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea585f02ba644e282726fbc1dde9b4a (DEFAULT_TABLE table=test-table [id=b2da57cfa7f541188031091f1be9da1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:31.940727 3642 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea585f02ba644e282726fbc1dde9b4a (DEFAULT_TABLE table=test-table [id=b2da57cfa7f541188031091f1be9da1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:31.940785 3920 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea585f02ba644e282726fbc1dde9b4a (DEFAULT_TABLE table=test-table [id=b2da57cfa7f541188031091f1be9da1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:31.957700 3642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea585f02ba644e282726fbc1dde9b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:31.957742 3785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea585f02ba644e282726fbc1dde9b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:31.961766 3920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea585f02ba644e282726fbc1dde9b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:31.964493 4102 tablet_bootstrap.cc:492] T 1ea585f02ba644e282726fbc1dde9b4a P ec96e0327eed44c3bf348feb2a47694a: Bootstrap starting.
I20260928 02:04:31.964972 4092 ts_tablet_manager.cc:1918] T 57b79cfdab654183ab7446bbc6c2e31f P 1ec164324ca148fcb9224f947a537cb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:04:31.965183 4092 log.cc:1196] T 57b79cfdab654183ab7446bbc6c2e31f P 1ec164324ca148fcb9224f947a537cb5: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/57b79cfdab654183ab7446bbc6c2e31f
I20260928 02:04:31.965731 4092 ts_tablet_manager.cc:1939] T 57b79cfdab654183ab7446bbc6c2e31f P 1ec164324ca148fcb9224f947a537cb5: Deleting consensus metadata
I20260928 02:04:31.966295 4102 tablet_bootstrap.cc:654] T 1ea585f02ba644e282726fbc1dde9b4a P ec96e0327eed44c3bf348feb2a47694a: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:31.968658 4102 tablet_bootstrap.cc:492] T 1ea585f02ba644e282726fbc1dde9b4a P ec96e0327eed44c3bf348feb2a47694a: No bootstrap required, opened a new log
I20260928 02:04:31.968847 4102 ts_tablet_manager.cc:1397] T 1ea585f02ba644e282726fbc1dde9b4a P ec96e0327eed44c3bf348feb2a47694a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20260928 02:04:31.968891 4103 tablet_bootstrap.cc:492] T 1ea585f02ba644e282726fbc1dde9b4a P 1ec164324ca148fcb9224f947a537cb5: Bootstrap starting.
I20260928 02:04:31.969554 4102 raft_consensus.cc:348] T 1ea585f02ba644e282726fbc1dde9b4a P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:31.969808 4102 raft_consensus.cc:374] T 1ea585f02ba644e282726fbc1dde9b4a P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:31.969885 4102 raft_consensus.cc:729] T 1ea585f02ba644e282726fbc1dde9b4a P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec96e0327eed44c3bf348feb2a47694a, State: Initialized, Role: FOLLOWER
I20260928 02:04:31.970146 4102 consensus_queue.cc:260] T 1ea585f02ba644e282726fbc1dde9b4a P ec96e0327eed44c3bf348feb2a47694a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:31.970577 4103 tablet_bootstrap.cc:654] T 1ea585f02ba644e282726fbc1dde9b4a P 1ec164324ca148fcb9224f947a537cb5: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:31.970697 4102 ts_tablet_manager.cc:1428] T 1ea585f02ba644e282726fbc1dde9b4a P ec96e0327eed44c3bf348feb2a47694a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:04:31.972595 4103 tablet_bootstrap.cc:492] T 1ea585f02ba644e282726fbc1dde9b4a P 1ec164324ca148fcb9224f947a537cb5: No bootstrap required, opened a new log
I20260928 02:04:31.972785 4103 ts_tablet_manager.cc:1397] T 1ea585f02ba644e282726fbc1dde9b4a P 1ec164324ca148fcb9224f947a537cb5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 02:04:31.974012 4103 raft_consensus.cc:348] T 1ea585f02ba644e282726fbc1dde9b4a P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:31.974265 4103 raft_consensus.cc:374] T 1ea585f02ba644e282726fbc1dde9b4a P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:31.974363 4103 raft_consensus.cc:729] T 1ea585f02ba644e282726fbc1dde9b4a P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec164324ca148fcb9224f947a537cb5, State: Initialized, Role: FOLLOWER
I20260928 02:04:31.974632 4103 consensus_queue.cc:260] T 1ea585f02ba644e282726fbc1dde9b4a P 1ec164324ca148fcb9224f947a537cb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:31.975150 4103 ts_tablet_manager.cc:1428] T 1ea585f02ba644e282726fbc1dde9b4a P 1ec164324ca148fcb9224f947a537cb5: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260928 02:04:31.986490 3493 catalog_manager.cc:4841] TS 1ec164324ca148fcb9224f947a537cb5 (127.1.182.130:40627): tablet 57b79cfdab654183ab7446bbc6c2e31f (table test-table [id=bff6d7bfeadb427788ea0e311664b7f9]) successfully deleted
I20260928 02:04:31.991392 4091 ts_tablet_manager.cc:1918] T 57b79cfdab654183ab7446bbc6c2e31f P ec96e0327eed44c3bf348feb2a47694a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:04:31.991389 4093 ts_tablet_manager.cc:1918] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:04:31.991573 4091 log.cc:1196] T 57b79cfdab654183ab7446bbc6c2e31f P ec96e0327eed44c3bf348feb2a47694a: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-0/wal/wals/57b79cfdab654183ab7446bbc6c2e31f
I20260928 02:04:31.991570 4093 log.cc:1196] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/57b79cfdab654183ab7446bbc6c2e31f
I20260928 02:04:31.992046 4093 ts_tablet_manager.cc:1939] T 57b79cfdab654183ab7446bbc6c2e31f P 357aad85e8d64c6db2afcef44b4bb3b5: Deleting consensus metadata
I20260928 02:04:31.992053 4091 ts_tablet_manager.cc:1939] T 57b79cfdab654183ab7446bbc6c2e31f P ec96e0327eed44c3bf348feb2a47694a: Deleting consensus metadata
I20260928 02:04:31.993247 3491 catalog_manager.cc:4841] TS ec96e0327eed44c3bf348feb2a47694a (127.1.182.129:36093): tablet 57b79cfdab654183ab7446bbc6c2e31f (table test-table [id=bff6d7bfeadb427788ea0e311664b7f9]) successfully deleted
I20260928 02:04:31.993350 3493 catalog_manager.cc:4841] TS 357aad85e8d64c6db2afcef44b4bb3b5 (127.1.182.131:37609): tablet 57b79cfdab654183ab7446bbc6c2e31f (table test-table [id=bff6d7bfeadb427788ea0e311664b7f9]) successfully deleted
I20260928 02:04:31.994709 4104 tablet_bootstrap.cc:492] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5: Bootstrap starting.
I20260928 02:04:31.996076 4104 tablet_bootstrap.cc:654] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:31.998167 4104 tablet_bootstrap.cc:492] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5: No bootstrap required, opened a new log
I20260928 02:04:31.998328 4104 ts_tablet_manager.cc:1397] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 02:04:31.998891 4104 raft_consensus.cc:348] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:31.999114 4104 raft_consensus.cc:374] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:31.999198 4104 raft_consensus.cc:729] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357aad85e8d64c6db2afcef44b4bb3b5, State: Initialized, Role: FOLLOWER
I20260928 02:04:31.999452 4104 consensus_queue.cc:260] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:31.999969 4104 ts_tablet_manager.cc:1428] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:04:32.051985 4071 raft_consensus.cc:482] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:32.052269 4071 raft_consensus.cc:504] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:32.052951 4071 leader_election.cc:288] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec96e0327eed44c3bf348feb2a47694a (127.1.182.129:36093), 1ec164324ca148fcb9224f947a537cb5 (127.1.182.130:40627)
I20260928 02:04:32.053527 3663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea585f02ba644e282726fbc1dde9b4a" candidate_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec96e0327eed44c3bf348feb2a47694a" is_pre_election: true
I20260928 02:04:32.053609 3805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea585f02ba644e282726fbc1dde9b4a" candidate_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ec164324ca148fcb9224f947a537cb5" is_pre_election: true
I20260928 02:04:32.053828 3663 raft_consensus.cc:2454] T 1ea585f02ba644e282726fbc1dde9b4a P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 357aad85e8d64c6db2afcef44b4bb3b5 in term 0.
I20260928 02:04:32.053905 3805 raft_consensus.cc:2454] T 1ea585f02ba644e282726fbc1dde9b4a P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 357aad85e8d64c6db2afcef44b4bb3b5 in term 0.
I20260928 02:04:32.054306 3873 leader_election.cc:302] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357aad85e8d64c6db2afcef44b4bb3b5, ec96e0327eed44c3bf348feb2a47694a; no voters:
I20260928 02:04:32.054634 4071 raft_consensus.cc:2781] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:04:32.054721 4071 raft_consensus.cc:482] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:04:32.054863 4071 raft_consensus.cc:3037] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:32.116138 4071 raft_consensus.cc:504] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:32.116909 4071 leader_election.cc:288] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 [CANDIDATE]: Term 1 election: Requested vote from peers ec96e0327eed44c3bf348feb2a47694a (127.1.182.129:36093), 1ec164324ca148fcb9224f947a537cb5 (127.1.182.130:40627)
I20260928 02:04:32.117475 3805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea585f02ba644e282726fbc1dde9b4a" candidate_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ec164324ca148fcb9224f947a537cb5"
I20260928 02:04:32.117460 3663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea585f02ba644e282726fbc1dde9b4a" candidate_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec96e0327eed44c3bf348feb2a47694a"
I20260928 02:04:32.117712 3663 raft_consensus.cc:3037] T 1ea585f02ba644e282726fbc1dde9b4a P ec96e0327eed44c3bf348feb2a47694a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:32.117710 3805 raft_consensus.cc:3037] T 1ea585f02ba644e282726fbc1dde9b4a P 1ec164324ca148fcb9224f947a537cb5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:32.120301 3805 raft_consensus.cc:2454] T 1ea585f02ba644e282726fbc1dde9b4a P 1ec164324ca148fcb9224f947a537cb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357aad85e8d64c6db2afcef44b4bb3b5 in term 1.
I20260928 02:04:32.120836 3875 leader_election.cc:302] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ec164324ca148fcb9224f947a537cb5, 357aad85e8d64c6db2afcef44b4bb3b5; no voters:
I20260928 02:04:32.121114 4071 raft_consensus.cc:2781] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:32.121174 3663 raft_consensus.cc:2454] T 1ea585f02ba644e282726fbc1dde9b4a P ec96e0327eed44c3bf348feb2a47694a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357aad85e8d64c6db2afcef44b4bb3b5 in term 1.
I20260928 02:04:32.121297 4071 raft_consensus.cc:686] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 [term 1 LEADER]: Becoming Leader. State: Replica: 357aad85e8d64c6db2afcef44b4bb3b5, State: Running, Role: LEADER
I20260928 02:04:32.121557 4071 consensus_queue.cc:237] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } }
I20260928 02:04:32.124193 3504 catalog_manager.cc:5495] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 357aad85e8d64c6db2afcef44b4bb3b5 (127.1.182.131). New cstate: current_term: 1 leader_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "357aad85e8d64c6db2afcef44b4bb3b5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37609 } health_report { overall_health: HEALTHY } } }
I20260928 02:04:32.158679 3663 raft_consensus.cc:1261] T 1ea585f02ba644e282726fbc1dde9b4a P ec96e0327eed44c3bf348feb2a47694a [term 1 FOLLOWER]: Refusing update from remote peer 357aad85e8d64c6db2afcef44b4bb3b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:04:32.158649 3807 raft_consensus.cc:1261] T 1ea585f02ba644e282726fbc1dde9b4a P 1ec164324ca148fcb9224f947a537cb5 [term 1 FOLLOWER]: Refusing update from remote peer 357aad85e8d64c6db2afcef44b4bb3b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:04:32.159205 4079 consensus_queue.cc:1035] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec96e0327eed44c3bf348feb2a47694a" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 36093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:32.159533 4071 consensus_queue.cc:1035] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ec164324ca148fcb9224f947a537cb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:04:32.180349 4072 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:04:33.762156 3505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:04:33.762356 3505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:04:33.766263 3505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f72032c6bdad4992b9023be55432f25c: Sending DeleteTablet for 3 replicas of tablet 1ea585f02ba644e282726fbc1dde9b4a
I20260928 02:04:33.767123 3642 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea585f02ba644e282726fbc1dde9b4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:33 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:59664
I20260928 02:04:33.767162 3785 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea585f02ba644e282726fbc1dde9b4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:33 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:46204
I20260928 02:04:33.767309 3925 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea585f02ba644e282726fbc1dde9b4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:33 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:34368
I20260928 02:04:33.769469 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 3577
I20260928 02:04:33.779979 4145 tablet_replica.cc:330] T 1ea585f02ba644e282726fbc1dde9b4a P ec96e0327eed44c3bf348feb2a47694a: stopping tablet replica
I20260928 02:04:33.780474 4145 raft_consensus.cc:2229] T 1ea585f02ba644e282726fbc1dde9b4a P ec96e0327eed44c3bf348feb2a47694a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:04:33.780921 4145 raft_consensus.cc:2258] T 1ea585f02ba644e282726fbc1dde9b4a P ec96e0327eed44c3bf348feb2a47694a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:33.782236 4145 ts_tablet_manager.cc:1905] T 1ea585f02ba644e282726fbc1dde9b4a P ec96e0327eed44c3bf348feb2a47694a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:33.782423 4146 tablet_replica.cc:330] T 1ea585f02ba644e282726fbc1dde9b4a P 1ec164324ca148fcb9224f947a537cb5: stopping tablet replica
I20260928 02:04:33.782610 4146 raft_consensus.cc:2229] T 1ea585f02ba644e282726fbc1dde9b4a P 1ec164324ca148fcb9224f947a537cb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:04:33.782855 4146 raft_consensus.cc:2258] T 1ea585f02ba644e282726fbc1dde9b4a P 1ec164324ca148fcb9224f947a537cb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:33.783425 4147 tablet_replica.cc:330] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5: stopping tablet replica
I20260928 02:04:33.783628 4147 raft_consensus.cc:2229] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:04:33.783993 4147 raft_consensus.cc:2258] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:33.784828 4146 ts_tablet_manager.cc:1905] T 1ea585f02ba644e282726fbc1dde9b4a P 1ec164324ca148fcb9224f947a537cb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:33.787631 4147 ts_tablet_manager.cc:1905] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 02:04:33 dist-test-slave-23bp krb5kdc[3461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561071, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:04:33.800833 4145 ts_tablet_manager.cc:1918] T 1ea585f02ba644e282726fbc1dde9b4a P ec96e0327eed44c3bf348feb2a47694a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:04:33.801015 4145 log.cc:1196] T 1ea585f02ba644e282726fbc1dde9b4a P ec96e0327eed44c3bf348feb2a47694a: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-0/wal/wals/1ea585f02ba644e282726fbc1dde9b4a
I20260928 02:04:33.801494 4145 ts_tablet_manager.cc:1939] T 1ea585f02ba644e282726fbc1dde9b4a P ec96e0327eed44c3bf348feb2a47694a: Deleting consensus metadata
I20260928 02:04:33.807336 4149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:33.770904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59678 (local address 127.1.182.129:36093)
0928 02:04:33.787503 (+ 16599us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:33.787508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:33.787535 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:04:33.787593 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:33.787597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:33.787659 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:33.787899 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:33.789371 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:33.790722 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:33.793015 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:33.793839 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:33.793983 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:33.799637 (+ 5654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:33.799652 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:33.799671 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:33.799705 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:33.804155 (+ 4450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:33.804985 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:33.804993 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:33.804995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:33.805248 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:33.805622 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:33.805625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:33.805627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:33.806062 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:33.806439 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:33.806588 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 02:04:33.806818 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16460,"thread_start_us":16347,"threads_started":1}
W20260928 02:04:33.808212 3708 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59678
W20260928 02:04:33.809002 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 3577: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:04:33.809317 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 3577
I20260928 02:04:33.818149 4147 ts_tablet_manager.cc:1918] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:04:33.818307 4147 log.cc:1196] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/1ea585f02ba644e282726fbc1dde9b4a
I20260928 02:04:33.818701 4147 ts_tablet_manager.cc:1939] T 1ea585f02ba644e282726fbc1dde9b4a P 357aad85e8d64c6db2afcef44b4bb3b5: Deleting consensus metadata
I20260928 02:04:33.852383 3493 catalog_manager.cc:4841] TS 357aad85e8d64c6db2afcef44b4bb3b5 (127.1.182.131:37609): tablet 1ea585f02ba644e282726fbc1dde9b4a (table test-table [id=b2da57cfa7f541188031091f1be9da1a]) successfully deleted
I20260928 02:04:33.868045 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 3717
W20260928 02:04:33.868045 3491 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.182.129:36093 (error 108)
I20260928 02:04:33.870407 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:33.869540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:36093 (local address 127.0.0.1:59680)
0928 02:04:33.870077 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:33.870235 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:36093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":287,"threads_started":1}
W20260928 02:04:33.870600 3491 catalog_manager.cc:4584] TS ec96e0327eed44c3bf348feb2a47694a (127.1.182.129:36093): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ea585f02ba644e282726fbc1dde9b4a: Network error: Client connection negotiation failed: client connection to 127.1.182.129:36093: connect: Connection refused (error 111)
Sep 28 02:04:33 dist-test-slave-23bp krb5kdc[3461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561071, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.182.130@KRBTEST.COM
I20260928 02:04:33.891319 4146 ts_tablet_manager.cc:1918] T 1ea585f02ba644e282726fbc1dde9b4a P 1ec164324ca148fcb9224f947a537cb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:04:33.891491 4146 log.cc:1196] T 1ea585f02ba644e282726fbc1dde9b4a P 1ec164324ca148fcb9224f947a537cb5: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/1ea585f02ba644e282726fbc1dde9b4a
I20260928 02:04:33.891976 4146 ts_tablet_manager.cc:1939] T 1ea585f02ba644e282726fbc1dde9b4a P 1ec164324ca148fcb9224f947a537cb5: Deleting consensus metadata
I20260928 02:04:33.892900 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:33.868876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46222 (local address 127.1.182.130:40627)
0928 02:04:33.869413 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:33.869446 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:33.869461 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:04:33.869524 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:33.869528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:33.869591 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:33.869704 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:33.871121 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:33.871883 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:33.873181 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:33.873608 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:33.873735 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:33.886775 (+ 13040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:33.886791 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:33.886810 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:33.886851 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:33.889755 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:33.890538 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:33.890546 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:33.890549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:33.890697 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:33.891162 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:33.891170 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:33.891173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:33.891697 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:33.891984 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:33.892272 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 02:04:33.892498 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":246,"threads_started":1}
I20260928 02:04:33.893258 3493 catalog_manager.cc:4841] TS 1ec164324ca148fcb9224f947a537cb5 (127.1.182.130:40627): tablet 1ea585f02ba644e282726fbc1dde9b4a (table test-table [id=b2da57cfa7f541188031091f1be9da1a]) successfully deleted
W20260928 02:04:33.893690 3847 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46222
W20260928 02:04:33.894340 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 3717: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:04:33.894507 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 3717
I20260928 02:04:33.917109 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 3856
I20260928 02:04:33.924124 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:33.923656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:36093 (local address 127.0.0.1:59684)
0928 02:04:33.923855 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:33.923950 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:36093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 02:04:33.925122 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:33.924748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:36093 (local address 127.0.0.1:59694)
0928 02:04:33.924898 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:33.924991 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:36093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
Sep 28 02:04:33 dist-test-slave-23bp krb5kdc[3461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561071, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.182.131@KRBTEST.COM
I20260928 02:04:33.935549 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:33.917923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34412 (local address 127.1.182.131:37609)
0928 02:04:33.918470 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:33.918474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:33.918490 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:04:33.918542 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:33.918545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:33.918602 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:33.918705 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:33.920170 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:33.920979 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:33.922182 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:33.922635 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:33.922735 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:33.928625 (+ 5890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:33.928644 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:33.928667 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:33.928714 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:33.932581 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:33.933401 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:33.933409 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:33.933412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:33.933567 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:33.933976 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:33.933986 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:33.933988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:33.934367 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:33.934706 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:33.934966 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 02:04:33.935214 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":270,"threads_started":1}
W20260928 02:04:33.936296 3986 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34412
W20260928 02:04:33.936926 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 3856: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:04:33.937096 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 3856
I20260928 02:04:33.957532 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 3474
I20260928 02:04:33.958842 3546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:34.078008 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:34.077463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:36093 (local address 127.0.0.1:59704)
0928 02:04:34.077707 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:34.077819 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:36093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 02:04:34.078544 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 3474
2026-09-28T02:04:34Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8505 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:04:35 dist-test-slave-23bp krb5kdc[4157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:04:35 dist-test-slave-23bp krb5kdc[4157](info): set up 2 sockets
Sep 28 02:04:35 dist-test-slave-23bp krb5kdc[4157](info): commencing operation
krb5kdc: starting...
W20260928 02:04:36.986012 1754 mini_kdc.cc:121] Time spent starting KDC: real 2.861s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:04:37 dist-test-slave-23bp krb5kdc[4157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:04:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:04:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:04:37.174351 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:36165
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:36165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:37.518676 4170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:37.518975 4170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:37.519120 4170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:37.526633 4170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:37.526741 4170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:37.526786 4170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:37.526842 4170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:37.526889 4170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:37.536737 4170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:36165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:36165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:37.538038 4170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:37.539330 4170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:37.551913 4175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:37.552000 4176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:37.556001 4178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:37.556593 4170 server_base.cc:1034] running on GCE node
I20260928 02:04:37.557498 4170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:37.559003 4170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:37.560268 4170 hybrid_clock.cc:648] HybridClock initialized: now 1790561077560211 us; error 44 us; skew 500 ppm
I20260928 02:04:37.560647 4170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:37 dist-test-slave-23bp krb5kdc[4157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561077, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:37.563835 4170 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:04:37.565394 4170 webserver.cc:460] Webserver started at http://127.1.182.190:45821/ using document root <none> and password file <none>
I20260928 02:04:37.566077 4170 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:37.566174 4170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:37.566429 4170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:37.569432 4170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "345b3531856843b3a505f409957e3240"
format_stamp: "Formatted at 2026-09-28 02:04:37 on dist-test-slave-23bp"
server_key: "da4584bfec558b053be8fff53dee4e8c"
server_key_iv: "fa42d2b3d0cd67e68885185c138390b5"
server_key_version: "encryptionkey@0"
I20260928 02:04:37.570158 4170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "345b3531856843b3a505f409957e3240"
format_stamp: "Formatted at 2026-09-28 02:04:37 on dist-test-slave-23bp"
server_key: "da4584bfec558b053be8fff53dee4e8c"
server_key_iv: "fa42d2b3d0cd67e68885185c138390b5"
server_key_version: "encryptionkey@0"
I20260928 02:04:37.575378 4170 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 02:04:37.578752 4185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:37.580191 4170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:04:37.580330 4170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "345b3531856843b3a505f409957e3240"
format_stamp: "Formatted at 2026-09-28 02:04:37 on dist-test-slave-23bp"
server_key: "da4584bfec558b053be8fff53dee4e8c"
server_key_iv: "fa42d2b3d0cd67e68885185c138390b5"
server_key_version: "encryptionkey@0"
I20260928 02:04:37.580489 4170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:37.600574 4170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:37.604542 4170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:37.604795 4170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:37.623303 4170 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:36165
I20260928 02:04:37.623335 4236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:36165 every 8 connection(s)
I20260928 02:04:37.624724 4170 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:04:37.628572 4237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:37.633725 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 4170
I20260928 02:04:37.633947 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:04:37.634699 1754 external_mini_cluster.cc:1428] Setting key f06fae95c67fa12f11c2d5df17c464a6
I20260928 02:04:37.640223 4237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240: Bootstrap starting.
I20260928 02:04:37.644011 4237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:37.645273 4237 log.cc:824] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:37.648799 4237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240: No bootstrap required, opened a new log
Sep 28 02:04:37 dist-test-slave-23bp krb5kdc[4157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:37.659531 4240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:37.636873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.1.182.190:36165)
0928 02:04:37.637744 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:37.637756 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:37.637798 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:04:37.639568 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:37.639593 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:37.639989 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:37.640375 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:37.642103 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:37.643437 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:37.645290 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:37.646277 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:37.646477 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:37.650836 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:37.650846 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:37.650867 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:37.650920 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:37.654910 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:37.655662 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:37.655671 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:37.655681 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:37.655890 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:37.656225 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:37.656230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:37.656232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:37.656971 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:37.657276 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:37.657597 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 02:04:37.658085 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":397,"threads_started":1}
I20260928 02:04:37.659783 4237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345b3531856843b3a505f409957e3240" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36165 } }
I20260928 02:04:37.660240 4237 raft_consensus.cc:374] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:37.660362 4237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345b3531856843b3a505f409957e3240, State: Initialized, Role: FOLLOWER
I20260928 02:04:37.660930 4237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345b3531856843b3a505f409957e3240" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36165 } }
I20260928 02:04:37.661196 4237 raft_consensus.cc:388] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:04:37.661345 4237 raft_consensus.cc:482] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:04:37.661535 4237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:37.663970 4237 raft_consensus.cc:504] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345b3531856843b3a505f409957e3240" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36165 } }
I20260928 02:04:37.664476 4237 leader_election.cc:302] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 345b3531856843b3a505f409957e3240; no voters:
I20260928 02:04:37.665052 4237 leader_election.cc:288] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:04:37.665200 4242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:37.665863 4242 raft_consensus.cc:686] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240 [term 1 LEADER]: Becoming Leader. State: Replica: 345b3531856843b3a505f409957e3240, State: Running, Role: LEADER
I20260928 02:04:37.666414 4242 consensus_queue.cc:237] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345b3531856843b3a505f409957e3240" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36165 } }
I20260928 02:04:37.667200 4237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:37.669265 4243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "345b3531856843b3a505f409957e3240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345b3531856843b3a505f409957e3240" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36165 } } }
I20260928 02:04:37.669605 4243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:37.670436 4247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:37.670291 4244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 345b3531856843b3a505f409957e3240. Latest consensus state: current_term: 1 leader_uuid: "345b3531856843b3a505f409957e3240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345b3531856843b3a505f409957e3240" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36165 } } }
I20260928 02:04:37.670650 4244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:37.676136 4247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:37.685737 4247 catalog_manager.cc:1347] Generated new cluster ID: dfe9f5af22d64fb8b9f6541f42c85ce9
I20260928 02:04:37.685859 4247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:37.707772 4247 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:04:37.709299 4247 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:37.722676 4247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 345b3531856843b3a505f409957e3240: Generated new TSK 0
I20260928 02:04:37.723654 4247 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.182.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
I20260928 02:04:37.779528 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:36165
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:38.123296 4263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:38.123580 4263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:38.123749 4263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:38.131667 4263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:38.131772 4263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:38.131944 4263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:04:38.141708 4263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:36165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:38.142987 4263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:38.144374 4263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:38.157374 4268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:38.157403 4269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:38.157321 4271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:38.158541 4263 server_base.cc:1034] running on GCE node
I20260928 02:04:38.159191 4263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:38.160175 4263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:38.161386 4263 hybrid_clock.cc:648] HybridClock initialized: now 1790561078161327 us; error 55 us; skew 500 ppm
I20260928 02:04:38.161700 4263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:38 dist-test-slave-23bp krb5kdc[4157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561078, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:38.164721 4263 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:04:38.166117 4263 webserver.cc:460] Webserver started at http://127.1.182.129:41113/ using document root <none> and password file <none>
I20260928 02:04:38.166831 4263 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:38.166970 4263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:38.167343 4263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:38.170609 4263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "db2a4edde7334fc68b3b1b9a15d6bc43"
format_stamp: "Formatted at 2026-09-28 02:04:38 on dist-test-slave-23bp"
server_key: "4734b41ddb98141e2a64f2f92edf883e"
server_key_iv: "59b5449517dd4d32f1fca873e6376d48"
server_key_version: "encryptionkey@0"
I20260928 02:04:38.171425 4263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "db2a4edde7334fc68b3b1b9a15d6bc43"
format_stamp: "Formatted at 2026-09-28 02:04:38 on dist-test-slave-23bp"
server_key: "4734b41ddb98141e2a64f2f92edf883e"
server_key_iv: "59b5449517dd4d32f1fca873e6376d48"
server_key_version: "encryptionkey@0"
I20260928 02:04:38.176312 4263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:04:38.179626 4278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:38.180896 4263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 02:04:38.181058 4263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "db2a4edde7334fc68b3b1b9a15d6bc43"
format_stamp: "Formatted at 2026-09-28 02:04:38 on dist-test-slave-23bp"
server_key: "4734b41ddb98141e2a64f2f92edf883e"
server_key_iv: "59b5449517dd4d32f1fca873e6376d48"
server_key_version: "encryptionkey@0"
I20260928 02:04:38.181262 4263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:38.194824 4263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:38.198891 4263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:38.199254 4263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:38.200189 4263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:38.201478 4263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:38.201582 4263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:38.201735 4263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:38.201813 4263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:38.234053 4263 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:45983
I20260928 02:04:38.234102 4390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:45983 every 8 connection(s)
I20260928 02:04:38.235411 4263 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:04:38.241415 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 4263
I20260928 02:04:38.241636 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:04:38.242446 1754 external_mini_cluster.cc:1428] Setting key 6d1e9e37f1b23e34004ed8d304f5a214
Sep 28 02:04:38 dist-test-slave-23bp krb5kdc[4157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561078, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:38.259171 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:38.237993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:44821 (local address 127.1.182.190:36165)
0928 02:04:38.238395 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:38.238400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:38.239536 (+ 1136us) server_negotiation.cc:407] Connection header received
0928 02:04:38.241744 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:38.241750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:38.241827 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:38.241915 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:38.243896 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:38.244832 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:38.247426 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:38.247811 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:38.247905 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:38.252293 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:38.252302 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:38.252310 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:38.252354 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:38.255908 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:38.256592 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:38.256600 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:38.256606 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:38.256747 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:38.257158 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:38.257166 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:38.257172 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:38.257541 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:38.257779 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:38.258366 (+ 587us) server_negotiation.cc:299] Negotiation successful
0928 02:04:38.258696 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":210,"threads_started":1}
I20260928 02:04:38.259871 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:38.238581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:36165 (local address 127.1.182.129:44821)
0928 02:04:38.239349 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:38.239406 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:38.241367 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:38.242203 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:38.242218 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:38.242964 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:38.243736 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:38.243754 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:38.244991 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:38.244997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:38.247292 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:38.247308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:38.248036 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:38.248046 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:38.248372 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:38.249123 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:38.249157 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:38.252030 (+ 2873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:38.256045 (+ 4015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:38.256052 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:38.256065 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:38.256459 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:38.256850 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:38.256854 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:38.256856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:38.257025 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:38.257660 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:38.257671 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:38.257970 (+ 299us) client_negotiation.cc:769] Sending connection context
0928 02:04:38.258184 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 02:04:38.258562 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":279,"threads_started":1}
I20260928 02:04:38.261404 4391 heartbeater.cc:347] Connected to a master server at 127.1.182.190:36165
I20260928 02:04:38.261723 4391 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.182.130@KRBTEST.COM; defaulting to no policy
I20260928 02:04:38.262411 4391 heartbeater.cc:511] Master 127.1.182.190:36165 requested a full tablet report, sending...
I20260928 02:04:38.264565 4202 ts_manager.cc:194] Registered new tserver with Master: db2a4edde7334fc68b3b1b9a15d6bc43 (127.1.182.129:45983)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.130@KRBTEST.COM" created.
I20260928 02:04:38.266026 4202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:44821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
I20260928 02:04:38.286643 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:36165
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:38.620522 4397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:38.620820 4397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:38.620910 4397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:38.631382 4397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:38.631539 4397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:38.631722 4397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:04:38.645746 4397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:36165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:38.646957 4397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:38.648341 4397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:38.662139 4402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:38.662077 4403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:38.662192 4405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:38.664747 4397 server_base.cc:1034] running on GCE node
I20260928 02:04:38.665351 4397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:38.666293 4397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:38.667511 4397 hybrid_clock.cc:648] HybridClock initialized: now 1790561078667450 us; error 59 us; skew 500 ppm
I20260928 02:04:38.667881 4397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:38 dist-test-slave-23bp krb5kdc[4157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561078, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:38.671005 4397 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:04:38.672437 4397 webserver.cc:460] Webserver started at http://127.1.182.130:40543/ using document root <none> and password file <none>
I20260928 02:04:38.673151 4397 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:38.673297 4397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:38.673637 4397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:38.677003 4397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "7408229f4f8a4f60a712a76024849039"
format_stamp: "Formatted at 2026-09-28 02:04:38 on dist-test-slave-23bp"
server_key: "b8721f720bc44a2318428d8c44786c2e"
server_key_iv: "1331a84489e2dcbb408c2a3c318a7771"
server_key_version: "encryptionkey@0"
I20260928 02:04:38.677793 4397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "7408229f4f8a4f60a712a76024849039"
format_stamp: "Formatted at 2026-09-28 02:04:38 on dist-test-slave-23bp"
server_key: "b8721f720bc44a2318428d8c44786c2e"
server_key_iv: "1331a84489e2dcbb408c2a3c318a7771"
server_key_version: "encryptionkey@0"
I20260928 02:04:38.683018 4397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:04:38.686838 4412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:38.688186 4397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:04:38.688359 4397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "7408229f4f8a4f60a712a76024849039"
format_stamp: "Formatted at 2026-09-28 02:04:38 on dist-test-slave-23bp"
server_key: "b8721f720bc44a2318428d8c44786c2e"
server_key_iv: "1331a84489e2dcbb408c2a3c318a7771"
server_key_version: "encryptionkey@0"
I20260928 02:04:38.688565 4397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:38.705112 4397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:38.709457 4397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:38.709758 4397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:38.710729 4397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:38.712076 4397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:38.712183 4397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:38.712339 4397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:38.712414 4397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:38.744845 4397 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:39801
I20260928 02:04:38.744882 4524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:39801 every 8 connection(s)
I20260928 02:04:38.746171 4397 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:04:38.755687 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 4397
I20260928 02:04:38.755888 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:04:38.756311 1754 external_mini_cluster.cc:1428] Setting key 9258355821ee60093268a7a66e524604
Sep 28 02:04:38 dist-test-slave-23bp krb5kdc[4157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561078, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:38.765637 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:38.748703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:58377 (local address 127.1.182.190:36165)
0928 02:04:38.748868 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:38.748872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:38.749971 (+ 1099us) server_negotiation.cc:407] Connection header received
0928 02:04:38.751929 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:38.751935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:38.752032 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:38.752157 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:38.753621 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:38.754324 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:38.756027 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:38.756483 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:38.756612 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:38.760127 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:38.760132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:38.760135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:38.760164 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:38.762671 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:38.763435 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:38.763445 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:38.763447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:38.763561 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:38.763977 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:38.763981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:38.763984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:38.764251 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:38.764401 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:38.765069 (+ 668us) server_negotiation.cc:299] Negotiation successful
0928 02:04:38.765290 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:04:38.766625 4527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:38.749135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:36165 (local address 127.1.182.130:58377)
0928 02:04:38.749819 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:38.749855 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:38.751627 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:38.752412 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:38.752420 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:38.752914 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:38.753458 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:38.753474 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:38.754484 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:38.754494 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:38.755862 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:38.755874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:38.756706 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:38.756725 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:38.756978 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:38.757715 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:38.757763 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:38.759905 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:38.762807 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:38.762815 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:38.762832 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:38.763314 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:38.763670 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:38.763674 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:38.763678 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:38.763860 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:38.764374 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:38.764382 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:38.764728 (+ 346us) client_negotiation.cc:769] Sending connection context
0928 02:04:38.764985 (+ 257us) client_negotiation.cc:240] Negotiation successful
0928 02:04:38.765504 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":250,"threads_started":1}
I20260928 02:04:38.768296 4525 heartbeater.cc:347] Connected to a master server at 127.1.182.190:36165
I20260928 02:04:38.768599 4525 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:38.769320 4525 heartbeater.cc:511] Master 127.1.182.190:36165 requested a full tablet report, sending...
I20260928 02:04:38.770901 4202 ts_manager.cc:194] Registered new tserver with Master: 7408229f4f8a4f60a712a76024849039 (127.1.182.130:39801)
I20260928 02:04:38.772178 4202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:58377
WARNING: no policy specified for kudu/127.1.182.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
I20260928 02:04:38.799420 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:36165
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:39.119930 4530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:39.120213 4530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:39.120306 4530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:39.127647 4530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:39.127777 4530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:39.127893 4530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:04:39.137758 4530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:36165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:39.139029 4530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:39.140314 4530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:39.153355 4535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:39.153767 4536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:39.154155 4538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:39.154156 4530 server_base.cc:1034] running on GCE node
I20260928 02:04:39.154824 4530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:39.155840 4530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:39.157073 4530 hybrid_clock.cc:648] HybridClock initialized: now 1790561079157033 us; error 33 us; skew 500 ppm
I20260928 02:04:39.157363 4530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:39 dist-test-slave-23bp krb5kdc[4157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561079, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:39.164247 4530 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:04:39.165789 4530 webserver.cc:460] Webserver started at http://127.1.182.131:39541/ using document root <none> and password file <none>
I20260928 02:04:39.166574 4530 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:39.166708 4530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:39.167038 4530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:39.170094 4530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "ab76663e21a941f4874af0392b507853"
format_stamp: "Formatted at 2026-09-28 02:04:39 on dist-test-slave-23bp"
server_key: "8d5106d67971facc8e056be0e40fbd3b"
server_key_iv: "8c21f0c2fee2e52900f17082cf36ec3a"
server_key_version: "encryptionkey@0"
I20260928 02:04:39.170804 4530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "ab76663e21a941f4874af0392b507853"
format_stamp: "Formatted at 2026-09-28 02:04:39 on dist-test-slave-23bp"
server_key: "8d5106d67971facc8e056be0e40fbd3b"
server_key_iv: "8c21f0c2fee2e52900f17082cf36ec3a"
server_key_version: "encryptionkey@0"
I20260928 02:04:39.176219 4530 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260928 02:04:39.179549 4545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:39.180858 4530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 02:04:39.181008 4530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "ab76663e21a941f4874af0392b507853"
format_stamp: "Formatted at 2026-09-28 02:04:39 on dist-test-slave-23bp"
server_key: "8d5106d67971facc8e056be0e40fbd3b"
server_key_iv: "8c21f0c2fee2e52900f17082cf36ec3a"
server_key_version: "encryptionkey@0"
I20260928 02:04:39.181206 4530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:39.196310 4530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:39.200805 4530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:39.201124 4530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:39.202126 4530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:39.203540 4530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:39.203612 4530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:39.203739 4530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:39.203801 4530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:39.237005 4530 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:34217
I20260928 02:04:39.237037 4657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:34217 every 8 connection(s)
I20260928 02:04:39.238276 4530 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:04:39.247761 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 4530
I20260928 02:04:39.247964 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:04:39.248453 1754 external_mini_cluster.cc:1428] Setting key a77b2cfc535bd0e6a42f41cace259711
Sep 28 02:04:39 dist-test-slave-23bp krb5kdc[4157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561079, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:39.258515 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:39.241043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:58079 (local address 127.1.182.190:36165)
0928 02:04:39.241242 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:39.241248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:39.242370 (+ 1122us) server_negotiation.cc:407] Connection header received
0928 02:04:39.244254 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:39.244258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:39.244321 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:39.244414 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:39.245862 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:39.246578 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:39.248356 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:39.248996 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:39.249095 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:39.253251 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:39.253255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:39.253258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:39.253288 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:39.255963 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:39.256587 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:39.256590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:39.256593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:39.256733 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:39.257123 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:39.257130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:39.257134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:39.257390 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:39.257495 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:39.258121 (+ 626us) server_negotiation.cc:299] Negotiation successful
0928 02:04:39.258271 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:04:39.259531 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:39.241495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:36165 (local address 127.1.182.131:58079)
0928 02:04:39.242199 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:39.242240 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:39.244000 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:39.244664 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:39.244672 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:39.245165 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:39.245718 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:39.245735 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:39.246719 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:39.246724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:39.248220 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:39.248231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:39.249227 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:39.249234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:39.249490 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:39.250365 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:39.250398 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:39.253074 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:39.256085 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:39.256092 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:39.256104 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:39.256483 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:39.256836 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:39.256839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:39.256842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:39.257009 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:39.257494 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:39.257499 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:39.257809 (+ 310us) client_negotiation.cc:769] Sending connection context
0928 02:04:39.258021 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 02:04:39.258395 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":289,"threads_started":1}
I20260928 02:04:39.260854 4658 heartbeater.cc:347] Connected to a master server at 127.1.182.190:36165
I20260928 02:04:39.261139 4658 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:39.261828 4658 heartbeater.cc:511] Master 127.1.182.190:36165 requested a full tablet report, sending...
I20260928 02:04:39.263518 4202 ts_manager.cc:194] Registered new tserver with Master: ab76663e21a941f4874af0392b507853 (127.1.182.131:34217)
I20260928 02:04:39.264482 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:04:39.264801 4202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:58079
I20260928 02:04:39.268996 4391 heartbeater.cc:503] Master 127.1.182.190:36165 was elected leader, sending a full tablet report...
W20260928 02:04:39.296799 4393 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:04:39.294396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48364 (local address 127.1.182.190:36165)
0928 02:04:39.294557 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:39.294562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:39.295123 (+ 561us) server_negotiation.cc:407] Connection header received
0928 02:04:39.295291 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:39.295295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:39.295356 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:39.295463 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:39.296585 (+ 1122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48364: BlockingRecv error: recv got EOF from 127.0.0.1:48364 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:04:39.300017 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 4263
W20260928 02:04:39.302915 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 4263: Not authorized: Client connection negotiation failed: client connection to 127.1.182.129:45983: server requires authentication, but client does not have Kerberos credentials available
I20260928 02:04:39.303099 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 4263
I20260928 02:04:39.318992 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 4397
W20260928 02:04:39.321539 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 4397: Not authorized: Client connection negotiation failed: client connection to 127.1.182.130:39801: server requires authentication, but client does not have Kerberos credentials available
I20260928 02:04:39.321743 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 4397
I20260928 02:04:39.336874 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 4530
W20260928 02:04:39.339792 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 4530: Not authorized: Client connection negotiation failed: client connection to 127.1.182.131:34217: server requires authentication, but client does not have Kerberos credentials available
I20260928 02:04:39.339959 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 4530
I20260928 02:04:39.355646 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 4170
I20260928 02:04:39.356801 4232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:39.460894 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 4170
2026-09-28T02:04:39Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5377 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:04:39 dist-test-slave-23bp krb5kdc[4675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:04:39 dist-test-slave-23bp krb5kdc[4675](info): set up 2 sockets
Sep 28 02:04:39 dist-test-slave-23bp krb5kdc[4675](info): commencing operation
krb5kdc: starting...
W20260928 02:04:40.562134 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:04:40 dist-test-slave-23bp krb5kdc[4675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:04:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:04:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:04:40.758353 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:45647
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:45647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:41.073521 4688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:41.073894 4688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:41.074047 4688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:41.081180 4688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:41.081344 4688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:41.081458 4688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:41.081563 4688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:41.081660 4688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:41.091571 4688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:45647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:45647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:41.092916 4688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:41.094404 4688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:41.106282 4696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:41.106241 4694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:41.106319 4693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:41.106830 4688 server_base.cc:1034] running on GCE node
I20260928 02:04:41.107796 4688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:41.109215 4688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:41.110446 4688 hybrid_clock.cc:648] HybridClock initialized: now 1790561081110403 us; error 55 us; skew 500 ppm
I20260928 02:04:41.110809 4688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:41 dist-test-slave-23bp krb5kdc[4675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561081, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:41.114027 4688 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:04:41.115502 4688 webserver.cc:460] Webserver started at http://127.1.182.190:32977/ using document root <none> and password file <none>
I20260928 02:04:41.116178 4688 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:41.116322 4688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:41.116634 4688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:41.119622 4688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "fee1442c56014fc8b1e3679829f94f0c"
format_stamp: "Formatted at 2026-09-28 02:04:41 on dist-test-slave-23bp"
server_key: "9aae2bfc83d8f1fcfb149d769b92b648"
server_key_iv: "b9b6ff0e8bab50cb24760ef5328c3be6"
server_key_version: "encryptionkey@0"
I20260928 02:04:41.120391 4688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "fee1442c56014fc8b1e3679829f94f0c"
format_stamp: "Formatted at 2026-09-28 02:04:41 on dist-test-slave-23bp"
server_key: "9aae2bfc83d8f1fcfb149d769b92b648"
server_key_iv: "b9b6ff0e8bab50cb24760ef5328c3be6"
server_key_version: "encryptionkey@0"
I20260928 02:04:41.125372 4688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 02:04:41.128723 4703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:41.130029 4688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:04:41.130216 4688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "fee1442c56014fc8b1e3679829f94f0c"
format_stamp: "Formatted at 2026-09-28 02:04:41 on dist-test-slave-23bp"
server_key: "9aae2bfc83d8f1fcfb149d769b92b648"
server_key_iv: "b9b6ff0e8bab50cb24760ef5328c3be6"
server_key_version: "encryptionkey@0"
I20260928 02:04:41.130425 4688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:41.149019 4688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:41.153290 4688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:41.153566 4688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:41.171883 4688 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:45647
I20260928 02:04:41.171916 4754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:45647 every 8 connection(s)
I20260928 02:04:41.173240 4688 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:04:41.176191 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 4688
I20260928 02:04:41.176394 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:04:41.176895 4755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:41.177088 1754 external_mini_cluster.cc:1428] Setting key b08401d6a9f2dbd6d13eb75cb1b89c62
I20260928 02:04:41.190392 4755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c: Bootstrap starting.
Sep 28 02:04:41 dist-test-slave-23bp krb5kdc[4675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:41.193434 4755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:41.194468 4755 log.cc:824] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:41.197364 4755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c: No bootstrap required, opened a new log
I20260928 02:04:41.200116 4758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:41.178908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.1.182.190:45647)
0928 02:04:41.179865 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:41.179881 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:41.179932 (+ 51us) server_negotiation.cc:407] Connection header received
0928 02:04:41.181756 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:41.181782 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:41.182292 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:41.182719 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:41.184057 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:41.185960 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:41.187400 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:41.188372 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:41.188575 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:41.192480 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:41.192488 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:41.192502 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:41.192537 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:41.195823 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:41.196552 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:41.196559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:41.196568 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:41.196723 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:41.197213 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:41.197218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:41.197222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:41.197840 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:41.198168 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:41.198587 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 02:04:41.199010 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":356,"threads_started":1}
I20260928 02:04:41.204396 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:04:41.206009 4755 raft_consensus.cc:348] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee1442c56014fc8b1e3679829f94f0c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 45647 } }
I20260928 02:04:41.206374 4755 raft_consensus.cc:374] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:41.206485 4755 raft_consensus.cc:729] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee1442c56014fc8b1e3679829f94f0c, State: Initialized, Role: FOLLOWER
I20260928 02:04:41.207017 4755 consensus_queue.cc:260] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee1442c56014fc8b1e3679829f94f0c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 45647 } }
I20260928 02:04:41.207299 4755 raft_consensus.cc:388] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:04:41.207437 4755 raft_consensus.cc:482] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:04:41.207636 4755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:41.209755 4755 raft_consensus.cc:504] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee1442c56014fc8b1e3679829f94f0c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 45647 } }
I20260928 02:04:41.210299 4755 leader_election.cc:302] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fee1442c56014fc8b1e3679829f94f0c; no voters:
I20260928 02:04:41.210983 4755 leader_election.cc:288] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:04:41.211126 4761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:41.213296 4755 sys_catalog.cc:564] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:41.218060 4761 raft_consensus.cc:686] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c [term 1 LEADER]: Becoming Leader. State: Replica: fee1442c56014fc8b1e3679829f94f0c, State: Running, Role: LEADER
I20260928 02:04:41.218750 4761 consensus_queue.cc:237] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee1442c56014fc8b1e3679829f94f0c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 45647 } }
I20260928 02:04:41.221920 4763 sys_catalog.cc:455] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fee1442c56014fc8b1e3679829f94f0c. Latest consensus state: current_term: 1 leader_uuid: "fee1442c56014fc8b1e3679829f94f0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee1442c56014fc8b1e3679829f94f0c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 45647 } } }
I20260928 02:04:41.222234 4763 sys_catalog.cc:458] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:41.221668 4762 sys_catalog.cc:455] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fee1442c56014fc8b1e3679829f94f0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee1442c56014fc8b1e3679829f94f0c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 45647 } } }
I20260928 02:04:41.222812 4762 sys_catalog.cc:458] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:41.230494 4771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:41.236196 4771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:41.243229 4780 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:04:41.246810 4771 catalog_manager.cc:1347] Generated new cluster ID: 16a0fa6f71c7450cba1062bdeb176867
I20260928 02:04:41.246941 4771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:41.258311 4758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:41.249145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33864 (local address 127.1.182.190:45647)
0928 02:04:41.249296 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:41.249302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:41.249972 (+ 670us) server_negotiation.cc:407] Connection header received
0928 02:04:41.250150 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:41.250156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:41.250240 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:41.250354 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:41.251521 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:41.252670 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:41.254412 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:41.255116 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:41.255247 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:41.256492 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:41.256499 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:41.256504 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:04:41.257417 (+ 913us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:41.257480 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:04:41.257515 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:41.257639 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:41.257795 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 02:04:41.257890 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":46}
I20260928 02:04:41.264189 4771 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:04:41.266453 4771 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:41.278573 4771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fee1442c56014fc8b1e3679829f94f0c: Generated new TSK 0
I20260928 02:04:41.279544 4771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:04:41.287442 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 4688
I20260928 02:04:41.288626 4750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:41.378407 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 4688
2026-09-28T02:04:41Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1913 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:04:41 dist-test-slave-23bp krb5kdc[4790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:04:41 dist-test-slave-23bp krb5kdc[4790](info): set up 2 sockets
Sep 28 02:04:41 dist-test-slave-23bp krb5kdc[4790](info): commencing operation
krb5kdc: starting...
W20260928 02:04:42.486256 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:04:42 dist-test-slave-23bp krb5kdc[4790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:04:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:04:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:04:42.676712 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:35363
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:35363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:42.992626 4803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:42.992928 4803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:42.993096 4803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:43.000335 4803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:43.000449 4803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:43.000573 4803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:43.000684 4803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:43.000748 4803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:43.010637 4803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:35363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:35363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:43.012061 4803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:43.013405 4803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:43.025251 4808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:43.025271 4811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:43.025207 4809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:43.026154 4803 server_base.cc:1034] running on GCE node
I20260928 02:04:43.026909 4803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:43.028347 4803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:43.029556 4803 hybrid_clock.cc:648] HybridClock initialized: now 1790561083029489 us; error 61 us; skew 500 ppm
I20260928 02:04:43.029891 4803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:43 dist-test-slave-23bp krb5kdc[4790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561083, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:43.033396 4803 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:04:43.034904 4803 webserver.cc:460] Webserver started at http://127.1.182.190:39429/ using document root <none> and password file <none>
I20260928 02:04:43.035650 4803 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:43.035790 4803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:43.036099 4803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:43.038975 4803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "8f07881cf8f4497299fe178287882e78"
format_stamp: "Formatted at 2026-09-28 02:04:43 on dist-test-slave-23bp"
server_key: "be61e0994e9fe7ba19479ee755140c93"
server_key_iv: "b105fea92e99014f85cb346ca9aa402c"
server_key_version: "encryptionkey@0"
I20260928 02:04:43.039780 4803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "8f07881cf8f4497299fe178287882e78"
format_stamp: "Formatted at 2026-09-28 02:04:43 on dist-test-slave-23bp"
server_key: "be61e0994e9fe7ba19479ee755140c93"
server_key_iv: "b105fea92e99014f85cb346ca9aa402c"
server_key_version: "encryptionkey@0"
I20260928 02:04:43.044945 4803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:04:43.048288 4818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:43.049611 4803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:04:43.049793 4803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "8f07881cf8f4497299fe178287882e78"
format_stamp: "Formatted at 2026-09-28 02:04:43 on dist-test-slave-23bp"
server_key: "be61e0994e9fe7ba19479ee755140c93"
server_key_iv: "b105fea92e99014f85cb346ca9aa402c"
server_key_version: "encryptionkey@0"
I20260928 02:04:43.049983 4803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:43.074543 4803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:43.078660 4803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:43.078955 4803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:43.097743 4803 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:35363
I20260928 02:04:43.097774 4869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:35363 every 8 connection(s)
I20260928 02:04:43.099188 4803 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:04:43.103029 4870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:43.104702 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 4803
I20260928 02:04:43.104915 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:04:43.105738 1754 external_mini_cluster.cc:1428] Setting key 944bcab364b5cd90336db4cd7f3e26b9
I20260928 02:04:43.118072 4870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78: Bootstrap starting.
Sep 28 02:04:43 dist-test-slave-23bp krb5kdc[4790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:43.121848 4870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:43.122936 4870 log.cc:824] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:43.126143 4870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78: No bootstrap required, opened a new log
I20260928 02:04:43.129087 4873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:43.107851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43788 (local address 127.1.182.190:35363)
0928 02:04:43.108655 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:43.108667 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:43.108707 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:04:43.110244 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:43.110265 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:43.110677 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:43.111003 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:43.112746 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:43.114064 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:43.115931 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:43.116625 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:43.116759 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:43.121477 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:43.121484 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:43.121499 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:43.121545 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:43.125039 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:43.125739 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:43.125745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:43.125752 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:43.125913 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:43.126301 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:43.126308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:43.126311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:43.126846 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:43.127270 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:43.127648 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 02:04:43.128073 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":381,"threads_started":1}
I20260928 02:04:43.134059 4870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f07881cf8f4497299fe178287882e78" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 35363 } }
I20260928 02:04:43.134433 4870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:43.134552 4870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f07881cf8f4497299fe178287882e78, State: Initialized, Role: FOLLOWER
I20260928 02:04:43.135219 4870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f07881cf8f4497299fe178287882e78" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 35363 } }
I20260928 02:04:43.135473 4870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:04:43.135619 4870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:04:43.135808 4870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:43.138393 4870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f07881cf8f4497299fe178287882e78" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 35363 } }
I20260928 02:04:43.138904 4870 leader_election.cc:302] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f07881cf8f4497299fe178287882e78; no voters:
I20260928 02:04:43.139488 4870 leader_election.cc:288] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:04:43.139671 4875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:43.140362 4875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78 [term 1 LEADER]: Becoming Leader. State: Replica: 8f07881cf8f4497299fe178287882e78, State: Running, Role: LEADER
I20260928 02:04:43.140887 4875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f07881cf8f4497299fe178287882e78" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 35363 } }
I20260928 02:04:43.141608 4870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:43.143868 4877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f07881cf8f4497299fe178287882e78. Latest consensus state: current_term: 1 leader_uuid: "8f07881cf8f4497299fe178287882e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f07881cf8f4497299fe178287882e78" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 35363 } } }
I20260928 02:04:43.144227 4877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:43.143962 4876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f07881cf8f4497299fe178287882e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f07881cf8f4497299fe178287882e78" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 35363 } } }
I20260928 02:04:43.144379 4876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:43.151589 4891 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:04:43.151757 4881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:43.157464 4881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:43.165784 4881 catalog_manager.cc:1347] Generated new cluster ID: 1d2e6a7cb7cf4b7dbaa6de5ee48a3709
I20260928 02:04:43.165906 4881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:43.175000 4881 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:04:43.176322 4881 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:43.188552 4881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f07881cf8f4497299fe178287882e78: Generated new TSK 0
I20260928 02:04:43.189419 4881 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.182.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
I20260928 02:04:43.238865 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:35363
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:43.571198 4896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:43.571564 4896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:43.571739 4896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:43.579370 4896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:43.579550 4896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:43.579744 4896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:04:43.590435 4896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:35363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:43.591879 4896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:43.593215 4896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:43.606251 4901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:43.606345 4904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:43.606355 4902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:43.607430 4896 server_base.cc:1034] running on GCE node
I20260928 02:04:43.608083 4896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:43.609018 4896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:43.610237 4896 hybrid_clock.cc:648] HybridClock initialized: now 1790561083610178 us; error 61 us; skew 500 ppm
I20260928 02:04:43.610551 4896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:43 dist-test-slave-23bp krb5kdc[4790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561083, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:43.613677 4896 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:04:43.615034 4896 webserver.cc:460] Webserver started at http://127.1.182.129:45797/ using document root <none> and password file <none>
I20260928 02:04:43.615749 4896 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:43.615846 4896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:43.616094 4896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:43.618849 4896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "02d584ddd3fc4ecb974beb7c1f145ec5"
format_stamp: "Formatted at 2026-09-28 02:04:43 on dist-test-slave-23bp"
server_key: "531ed7111afcf4938580ac541bc84dd5"
server_key_iv: "7fda6837b1d3c05f54de3586310ce782"
server_key_version: "encryptionkey@0"
I20260928 02:04:43.619647 4896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "02d584ddd3fc4ecb974beb7c1f145ec5"
format_stamp: "Formatted at 2026-09-28 02:04:43 on dist-test-slave-23bp"
server_key: "531ed7111afcf4938580ac541bc84dd5"
server_key_iv: "7fda6837b1d3c05f54de3586310ce782"
server_key_version: "encryptionkey@0"
I20260928 02:04:43.624691 4896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 02:04:43.628183 4911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:43.629652 4896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:04:43.629809 4896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "02d584ddd3fc4ecb974beb7c1f145ec5"
format_stamp: "Formatted at 2026-09-28 02:04:43 on dist-test-slave-23bp"
server_key: "531ed7111afcf4938580ac541bc84dd5"
server_key_iv: "7fda6837b1d3c05f54de3586310ce782"
server_key_version: "encryptionkey@0"
I20260928 02:04:43.629985 4896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:43.641361 4896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:43.645666 4896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:43.645956 4896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:43.646857 4896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:43.648234 4896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:43.648306 4896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:43.648465 4896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:43.648545 4896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:43.681938 4896 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:36637
I20260928 02:04:43.682042 5023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:36637 every 8 connection(s)
I20260928 02:04:43.683254 4896 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:04:43.689414 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 4896
I20260928 02:04:43.689618 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:04:43.690277 1754 external_mini_cluster.cc:1428] Setting key 7934fd3b30d6deb9afaa867e31e267ff
Sep 28 02:04:43 dist-test-slave-23bp krb5kdc[4790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561083, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:43.704152 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:43.685937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:52671 (local address 127.1.182.190:35363)
0928 02:04:43.686417 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:43.686422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:43.687334 (+ 912us) server_negotiation.cc:407] Connection header received
0928 02:04:43.689223 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:43.689228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:43.689318 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:43.689436 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:43.691037 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:43.692098 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:43.693661 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:43.694046 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:43.694174 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:43.697836 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:43.697843 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:43.697851 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:43.697894 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:43.700795 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:43.701728 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:43.701734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:43.701739 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:43.701868 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:43.702365 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:43.702369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:43.702371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:43.702653 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:43.702778 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:43.703703 (+ 925us) server_negotiation.cc:299] Negotiation successful
0928 02:04:43.703870 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":265,"threads_started":1}
I20260928 02:04:43.705763 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:43.686344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:35363 (local address 127.1.182.129:52671)
0928 02:04:43.687145 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:43.687207 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:43.688928 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:43.689682 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:43.689690 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:43.690256 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:43.690860 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:43.690877 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:43.692261 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:43.692266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:43.693496 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:43.693511 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:43.694269 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:43.694276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:43.694503 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:43.695224 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:43.695246 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:43.697619 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:43.700928 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:43.700940 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:43.700959 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:43.701591 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:43.701988 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:43.701995 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:43.702002 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:43.702250 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:43.702802 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:43.702813 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:43.703288 (+ 475us) client_negotiation.cc:769] Sending connection context
0928 02:04:43.703610 (+ 322us) client_negotiation.cc:240] Negotiation successful
0928 02:04:43.704209 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":283,"threads_started":1}
I20260928 02:04:43.707587 5024 heartbeater.cc:347] Connected to a master server at 127.1.182.190:35363
I20260928 02:04:43.707924 5024 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:43.708670 5024 heartbeater.cc:511] Master 127.1.182.190:35363 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.182.130@KRBTEST.COM; defaulting to no policy
I20260928 02:04:43.710808 4835 ts_manager.cc:194] Registered new tserver with Master: 02d584ddd3fc4ecb974beb7c1f145ec5 (127.1.182.129:36637)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.130@KRBTEST.COM" created.
I20260928 02:04:43.712544 4835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:52671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
I20260928 02:04:43.735033 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:35363
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:44.050935 5030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:44.051280 5030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:44.051437 5030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:44.058718 5030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:44.058825 5030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:44.059003 5030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:04:44.068919 5030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:35363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:44.070248 5030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:44.071581 5030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:44.085199 5036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:44.085253 5038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:44.085136 5035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:44.086148 5030 server_base.cc:1034] running on GCE node
I20260928 02:04:44.086781 5030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:44.087878 5030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:44.089108 5030 hybrid_clock.cc:648] HybridClock initialized: now 1790561084089062 us; error 44 us; skew 500 ppm
I20260928 02:04:44.089426 5030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:44 dist-test-slave-23bp krb5kdc[4790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561084, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:44.092589 5030 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:04:44.094090 5030 webserver.cc:460] Webserver started at http://127.1.182.130:43775/ using document root <none> and password file <none>
I20260928 02:04:44.094789 5030 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:44.094933 5030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:44.095293 5030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:44.098259 5030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "02a38af3834a456d874897d999cb8855"
format_stamp: "Formatted at 2026-09-28 02:04:44 on dist-test-slave-23bp"
server_key: "bc09833ecbc1797087dcaf40cca7f063"
server_key_iv: "2a93be0c3ca200a7a6a26988f416b507"
server_key_version: "encryptionkey@0"
I20260928 02:04:44.099031 5030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "02a38af3834a456d874897d999cb8855"
format_stamp: "Formatted at 2026-09-28 02:04:44 on dist-test-slave-23bp"
server_key: "bc09833ecbc1797087dcaf40cca7f063"
server_key_iv: "2a93be0c3ca200a7a6a26988f416b507"
server_key_version: "encryptionkey@0"
I20260928 02:04:44.104334 5030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:04:44.107973 5045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:44.109431 5030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:04:44.109596 5030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "02a38af3834a456d874897d999cb8855"
format_stamp: "Formatted at 2026-09-28 02:04:44 on dist-test-slave-23bp"
server_key: "bc09833ecbc1797087dcaf40cca7f063"
server_key_iv: "2a93be0c3ca200a7a6a26988f416b507"
server_key_version: "encryptionkey@0"
I20260928 02:04:44.109772 5030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:44.124169 5030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:44.128115 5030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:44.128409 5030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:44.129365 5030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:44.130715 5030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:44.130787 5030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:44.130944 5030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:44.131019 5030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:44.164872 5030 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:45009
I20260928 02:04:44.164927 5157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:45009 every 8 connection(s)
I20260928 02:04:44.166188 5030 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:04:44.173203 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 5030
I20260928 02:04:44.173516 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:04:44.174113 1754 external_mini_cluster.cc:1428] Setting key 9623a914e1eb535aadf6856ae68dda49
Sep 28 02:04:44 dist-test-slave-23bp krb5kdc[4790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561084, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:44.191116 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:44.168756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:40825 (local address 127.1.182.190:35363)
0928 02:04:44.168927 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:44.168931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:44.169996 (+ 1065us) server_negotiation.cc:407] Connection header received
0928 02:04:44.171774 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:44.171779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:44.171845 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:44.171933 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:44.173432 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:44.174526 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:44.176680 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:44.177371 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:44.177519 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:44.185298 (+ 7779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:44.185302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:44.185305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:44.185336 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:44.187954 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:44.188695 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:44.188704 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:44.188707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:44.188831 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:44.189342 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:44.189347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:44.189350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:44.189643 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:44.189800 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:44.190506 (+ 706us) server_negotiation.cc:299] Negotiation successful
0928 02:04:44.190741 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:04:44.192131 5160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:44.169142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:35363 (local address 127.1.182.130:40825)
0928 02:04:44.169829 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:44.169867 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:44.171531 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:44.172142 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:44.172149 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:44.172629 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:44.173188 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:44.173205 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:44.174691 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:44.174699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:44.176534 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:44.176545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:44.177566 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:44.177571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:44.177758 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:44.182347 (+ 4589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:44.182381 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:44.185049 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:44.188112 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:44.188121 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:44.188140 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:44.188563 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:44.189012 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:44.189018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:44.189024 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:44.189223 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:44.189781 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:44.189788 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:44.190147 (+ 359us) client_negotiation.cc:769] Sending connection context
0928 02:04:44.190418 (+ 271us) client_negotiation.cc:240] Negotiation successful
0928 02:04:44.190964 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":263,"threads_started":1}
WARNING: no policy specified for kudu/127.1.182.131@KRBTEST.COM; defaulting to no policy
I20260928 02:04:44.193722 5158 heartbeater.cc:347] Connected to a master server at 127.1.182.190:35363
I20260928 02:04:44.194028 5158 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:44.194777 5158 heartbeater.cc:511] Master 127.1.182.190:35363 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.131@KRBTEST.COM" created.
I20260928 02:04:44.196448 4835 ts_manager.cc:194] Registered new tserver with Master: 02a38af3834a456d874897d999cb8855 (127.1.182.130:45009)
I20260928 02:04:44.197690 4835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:40825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
I20260928 02:04:44.217629 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:35363
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:44.544493 5163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:44.544828 5163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:44.545004 5163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:44.552407 5163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:44.552584 5163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:44.552762 5163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:04:44.562709 5163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:35363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:44.564262 5163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:44.565660 5163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:44.578909 5169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:44.578936 5171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:44.579190 5168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:44.579640 5163 server_base.cc:1034] running on GCE node
I20260928 02:04:44.580201 5163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:44.581153 5163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:44.582341 5163 hybrid_clock.cc:648] HybridClock initialized: now 1790561084582290 us; error 47 us; skew 500 ppm
I20260928 02:04:44.582618 5163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:44 dist-test-slave-23bp krb5kdc[4790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561084, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:44.585978 5163 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:04:44.587514 5163 webserver.cc:460] Webserver started at http://127.1.182.131:43319/ using document root <none> and password file <none>
I20260928 02:04:44.588248 5163 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:44.588382 5163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:44.588732 5163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:44.591722 5163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "3dfe98818ebb4db8a9cc9a19dbb89708"
format_stamp: "Formatted at 2026-09-28 02:04:44 on dist-test-slave-23bp"
server_key: "f48d9e15afde5cca639e116e19cd05c1"
server_key_iv: "592776ee3cf1481b50ecfd67790ac715"
server_key_version: "encryptionkey@0"
I20260928 02:04:44.592530 5163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "3dfe98818ebb4db8a9cc9a19dbb89708"
format_stamp: "Formatted at 2026-09-28 02:04:44 on dist-test-slave-23bp"
server_key: "f48d9e15afde5cca639e116e19cd05c1"
server_key_iv: "592776ee3cf1481b50ecfd67790ac715"
server_key_version: "encryptionkey@0"
I20260928 02:04:44.597934 5163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:04:44.601569 5178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:44.603111 5163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:04:44.603260 5163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "3dfe98818ebb4db8a9cc9a19dbb89708"
format_stamp: "Formatted at 2026-09-28 02:04:44 on dist-test-slave-23bp"
server_key: "f48d9e15afde5cca639e116e19cd05c1"
server_key_iv: "592776ee3cf1481b50ecfd67790ac715"
server_key_version: "encryptionkey@0"
I20260928 02:04:44.603433 5163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:44.617177 5163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:44.621384 5163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:44.621640 5163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:44.622507 5163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:44.623899 5163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:44.624003 5163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:44.624150 5163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:44.624226 5163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:44.656678 5163 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:36431
I20260928 02:04:44.656759 5290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:36431 every 8 connection(s)
I20260928 02:04:44.657984 5163 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:04:44.665956 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 5163
I20260928 02:04:44.666147 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:04:44.666706 1754 external_mini_cluster.cc:1428] Setting key dea7b43f85f476e049b43b4433e72feb
Sep 28 02:04:44 dist-test-slave-23bp krb5kdc[4790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561084, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:44.678017 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:44.660871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:59329 (local address 127.1.182.190:35363)
0928 02:04:44.661077 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:44.661081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:44.662180 (+ 1099us) server_negotiation.cc:407] Connection header received
0928 02:04:44.664026 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:44.664033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:44.664128 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:44.664240 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:44.665758 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:44.666689 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:44.668676 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:44.669110 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:44.669242 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:44.672604 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:44.672608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:44.672612 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:44.672642 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:44.675205 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:44.675958 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:44.675961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:44.675963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:44.676086 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:44.676581 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:44.676584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:44.676586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:44.676837 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:44.676940 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:44.677615 (+ 675us) server_negotiation.cc:299] Negotiation successful
0928 02:04:44.677779 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:04:44.679057 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:44.661236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:35363 (local address 127.1.182.131:59329)
0928 02:04:44.661989 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:44.662047 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:44.663730 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:44.664492 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:44.664500 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:44.665000 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:44.665582 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:44.665598 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:44.666842 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:44.666848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:44.668535 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:44.668553 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:44.669329 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:44.669337 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:44.669592 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:44.670292 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:44.670313 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:44.672414 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:44.675348 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:44.675355 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:44.675369 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:44.675847 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:44.676211 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:44.676215 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:44.676217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:44.676414 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:44.676946 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:44.676953 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:44.677263 (+ 310us) client_negotiation.cc:769] Sending connection context
0928 02:04:44.677539 (+ 276us) client_negotiation.cc:240] Negotiation successful
0928 02:04:44.677921 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":289,"threads_started":1}
I20260928 02:04:44.680728 5291 heartbeater.cc:347] Connected to a master server at 127.1.182.190:35363
I20260928 02:04:44.681094 5291 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:44.682027 5291 heartbeater.cc:511] Master 127.1.182.190:35363 requested a full tablet report, sending...
I20260928 02:04:44.683552 4835 ts_manager.cc:194] Registered new tserver with Master: 3dfe98818ebb4db8a9cc9a19dbb89708 (127.1.182.131:36431)
I20260928 02:04:44.684675 4835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:59329
I20260928 02:04:44.694475 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:04:44.716365 5024 heartbeater.cc:503] Master 127.1.182.190:35363 was elected leader, sending a full tablet report...
Sep 28 02:04:44 dist-test-slave-23bp krb5kdc[4790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561084, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 02:04:44 dist-test-slave-23bp krb5kdc[4790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561084, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:44.743903 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:44.729307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.1.182.190:35363)
0928 02:04:44.729504 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:44.729509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:44.730114 (+ 605us) server_negotiation.cc:407] Connection header received
0928 02:04:44.730341 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:44.730346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:44.730420 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:44.730520 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:44.732092 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:44.732863 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:44.734369 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:44.734846 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:44.734967 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:44.738747 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:44.738752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:44.738755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:44.738795 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:44.741376 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:44.742083 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:44.742088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:44.742090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:44.742227 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:44.742591 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:44.742596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:44.742620 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:44.742981 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:44.743256 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:44.743512 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 02:04:44.743679 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 02:04:44.744472 4835 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43804
I20260928 02:04:44.746898 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 4896
Sep 28 02:04:44 dist-test-slave-23bp krb5kdc[4790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561084, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:04:44.763612 5302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:44.748015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.1.182.129:36637)
0928 02:04:44.748709 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:44.748717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:44.748741 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:04:44.748796 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:44.748802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:44.749015 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:44.749140 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:44.750451 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:44.751624 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:44.752850 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:44.753582 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:44.753686 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:44.757679 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:44.757691 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:44.757707 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:44.757749 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:44.760762 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:44.761528 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:44.761536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:44.761538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:44.761673 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:44.762031 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:44.762035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:44.762037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:44.762400 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:44.762627 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:44.762876 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 02:04:44.763168 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":374,"threads_started":1}
W20260928 02:04:44.764946 5019 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53552
W20260928 02:04:44.766215 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 4896: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:04:44.766439 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 4896
I20260928 02:04:44.782796 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 5030
Sep 28 02:04:44 dist-test-slave-23bp krb5kdc[4790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561084, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.182.130@KRBTEST.COM
I20260928 02:04:44.800001 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:44.783788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60268 (local address 127.1.182.130:45009)
0928 02:04:44.784387 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:44.784395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:44.784421 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:04:44.784482 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:44.784489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:44.784723 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:44.784853 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:44.786214 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:44.787312 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:44.788594 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:44.789130 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:44.789237 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:44.793717 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:44.793730 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:44.793746 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:44.793786 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:44.796952 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:44.797660 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:44.797668 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:44.797670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:44.797792 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:44.798204 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:44.798207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:44.798209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:44.798552 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:44.798787 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:44.799134 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 02:04:44.799434 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":332,"threads_started":1}
W20260928 02:04:44.801283 5153 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60268
W20260928 02:04:44.802367 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 5030: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:04:44.802564 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 5030
I20260928 02:04:44.817821 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 5163
Sep 28 02:04:44 dist-test-slave-23bp krb5kdc[4790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561084, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.182.131@KRBTEST.COM
I20260928 02:04:44.834635 5304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:44.818867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48364 (local address 127.1.182.131:36431)
0928 02:04:44.819534 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:44.819542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:44.819571 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:04:44.819631 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:44.819637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:44.819874 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:44.820098 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:44.821438 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:44.822482 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:44.823927 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:44.824484 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:44.824625 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:44.828894 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:44.828906 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:44.828921 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:44.828954 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:44.831809 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:44.832491 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:44.832498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:44.832500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:44.832621 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:44.833020 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:44.833024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:44.833026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:44.833355 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:44.833675 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:44.834008 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 02:04:44.834278 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":375,"threads_started":1}
W20260928 02:04:44.836184 5286 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48364
W20260928 02:04:44.837364 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 5163: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:04:44.837561 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 5163
I20260928 02:04:44.855188 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 4803
I20260928 02:04:44.856423 4865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:44.961302 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 4803
2026-09-28T02:04:44Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3587 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:04:45 dist-test-slave-23bp krb5kdc[5308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:04:45 dist-test-slave-23bp krb5kdc[5308](info): set up 2 sockets
Sep 28 02:04:45 dist-test-slave-23bp krb5kdc[5308](info): commencing operation
krb5kdc: starting...
W20260928 02:04:46.049969 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:04:46 dist-test-slave-23bp krb5kdc[5308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:04:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:04:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:04:46.259234 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:39775
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:33341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:39775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:46.584625 5321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:46.584913 5321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:46.585069 5321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:46.592278 5321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:46.592393 5321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:46.592479 5321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:46.592593 5321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:46.592680 5321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:46.602607 5321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:33341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:39775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:39775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:46.603962 5321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:46.605333 5321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:46.617118 5327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:46.617290 5329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:46.617677 5326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:46.618067 5321 server_base.cc:1034] running on GCE node
I20260928 02:04:46.618796 5321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:46.620240 5321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:46.621482 5321 hybrid_clock.cc:648] HybridClock initialized: now 1790561086621442 us; error 58 us; skew 500 ppm
I20260928 02:04:46.621868 5321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:46 dist-test-slave-23bp krb5kdc[5308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561086, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:46.625101 5321 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:04:46.626489 5321 webserver.cc:460] Webserver started at http://127.1.182.190:39057/ using document root <none> and password file <none>
I20260928 02:04:46.627357 5321 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:46.627501 5321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:46.627806 5321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:46.630715 5321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "4f81db9d5d2e44c3bb87bac4de3f4c66"
format_stamp: "Formatted at 2026-09-28 02:04:46 on dist-test-slave-23bp"
server_key: "afb172742272bd3f756906387285a546"
server_key_iv: "c3e52a9469eba2756d9be20a16a317b3"
server_key_version: "encryptionkey@0"
I20260928 02:04:46.631500 5321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "4f81db9d5d2e44c3bb87bac4de3f4c66"
format_stamp: "Formatted at 2026-09-28 02:04:46 on dist-test-slave-23bp"
server_key: "afb172742272bd3f756906387285a546"
server_key_iv: "c3e52a9469eba2756d9be20a16a317b3"
server_key_version: "encryptionkey@0"
I20260928 02:04:46.636725 5321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 02:04:46.640311 5336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:46.641681 5321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:04:46.641862 5321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "4f81db9d5d2e44c3bb87bac4de3f4c66"
format_stamp: "Formatted at 2026-09-28 02:04:46 on dist-test-slave-23bp"
server_key: "afb172742272bd3f756906387285a546"
server_key_iv: "c3e52a9469eba2756d9be20a16a317b3"
server_key_version: "encryptionkey@0"
I20260928 02:04:46.642069 5321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:46.656613 5321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:46.660809 5321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:46.661103 5321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:46.680034 5321 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:39775
I20260928 02:04:46.680071 5387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:39775 every 8 connection(s)
I20260928 02:04:46.681488 5321 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:04:46.685273 5388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:46.688309 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 5321
I20260928 02:04:46.688539 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:04:46.689347 1754 external_mini_cluster.cc:1428] Setting key 859b585e085897155f432c1258af8f6c
I20260928 02:04:46.697901 5388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66: Bootstrap starting.
I20260928 02:04:46.701692 5388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:46.703070 5388 log.cc:824] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:04:46 dist-test-slave-23bp krb5kdc[5308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:46.706250 5388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66: No bootstrap required, opened a new log
I20260928 02:04:46.713680 5391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:46.691470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45156 (local address 127.1.182.190:39775)
0928 02:04:46.692194 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:46.692205 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:46.692245 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:04:46.693870 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:46.693891 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:46.694296 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:46.694617 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:46.696454 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:46.697775 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:46.699559 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:46.700217 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:46.700343 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:46.705509 (+ 5166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:46.705517 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:46.705535 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:46.705575 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:46.708943 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:46.709810 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:46.709817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:46.709824 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:46.710054 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:46.710506 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:46.710509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:46.710549 (+ 40us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:46.711298 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:46.711543 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:46.712069 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 02:04:46.712552 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":334,"threads_started":1}
I20260928 02:04:46.715888 5388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f81db9d5d2e44c3bb87bac4de3f4c66" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39775 } }
I20260928 02:04:46.716310 5388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:46.716434 5388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f81db9d5d2e44c3bb87bac4de3f4c66, State: Initialized, Role: FOLLOWER
I20260928 02:04:46.717192 5388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f81db9d5d2e44c3bb87bac4de3f4c66" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39775 } }
I20260928 02:04:46.717460 5388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:04:46.717600 5388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:04:46.717803 5388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:46.720166 5388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f81db9d5d2e44c3bb87bac4de3f4c66" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39775 } }
I20260928 02:04:46.720702 5388 leader_election.cc:302] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f81db9d5d2e44c3bb87bac4de3f4c66; no voters:
I20260928 02:04:46.721395 5388 leader_election.cc:288] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:04:46.721547 5393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:46.722417 5393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66 [term 1 LEADER]: Becoming Leader. State: Replica: 4f81db9d5d2e44c3bb87bac4de3f4c66, State: Running, Role: LEADER
I20260928 02:04:46.722988 5393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f81db9d5d2e44c3bb87bac4de3f4c66" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39775 } }
I20260928 02:04:46.723654 5388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:46.726156 5395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f81db9d5d2e44c3bb87bac4de3f4c66. Latest consensus state: current_term: 1 leader_uuid: "4f81db9d5d2e44c3bb87bac4de3f4c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f81db9d5d2e44c3bb87bac4de3f4c66" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39775 } } }
I20260928 02:04:46.726657 5395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:46.726248 5394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f81db9d5d2e44c3bb87bac4de3f4c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f81db9d5d2e44c3bb87bac4de3f4c66" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39775 } } }
I20260928 02:04:46.726873 5394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:46.729311 5399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:46.734091 5399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:46.743378 5399 catalog_manager.cc:1347] Generated new cluster ID: 26019af9f34a4ef79e92714418e0916f
I20260928 02:04:46.743585 5399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:46.759121 5399 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:04:46.760993 5399 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:46.774434 5399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f81db9d5d2e44c3bb87bac4de3f4c66: Generated new TSK 0
I20260928 02:04:46.775580 5399 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.182.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
I20260928 02:04:46.827457 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39775
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:33341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:47.154526 5414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:47.154850 5414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:47.154987 5414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:47.162233 5414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:47.162335 5414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:47.162454 5414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:04:47.172117 5414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:33341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:47.173494 5414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:47.174779 5414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:47.188054 5420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:47.188211 5422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:47.188000 5419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:47.189715 5414 server_base.cc:1034] running on GCE node
I20260928 02:04:47.190328 5414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:47.191360 5414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:47.192669 5414 hybrid_clock.cc:648] HybridClock initialized: now 1790561087192633 us; error 94 us; skew 500 ppm
I20260928 02:04:47.192991 5414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:47 dist-test-slave-23bp krb5kdc[5308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:47.196283 5414 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:04:47.197726 5414 webserver.cc:460] Webserver started at http://127.1.182.129:33473/ using document root <none> and password file <none>
I20260928 02:04:47.198361 5414 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:47.198465 5414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:47.198715 5414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:47.201511 5414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "f55f021aabee481f9c8af32c89d0fc71"
format_stamp: "Formatted at 2026-09-28 02:04:47 on dist-test-slave-23bp"
server_key: "c7bdd08e9f71c91e13ea817211b106a6"
server_key_iv: "5da9e7bcff7a4d6be83dcbb5e5b4d7c0"
server_key_version: "encryptionkey@0"
I20260928 02:04:47.202301 5414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "f55f021aabee481f9c8af32c89d0fc71"
format_stamp: "Formatted at 2026-09-28 02:04:47 on dist-test-slave-23bp"
server_key: "c7bdd08e9f71c91e13ea817211b106a6"
server_key_iv: "5da9e7bcff7a4d6be83dcbb5e5b4d7c0"
server_key_version: "encryptionkey@0"
I20260928 02:04:47.207513 5414 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:04:47.211015 5429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:47.212601 5414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:04:47.212786 5414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "f55f021aabee481f9c8af32c89d0fc71"
format_stamp: "Formatted at 2026-09-28 02:04:47 on dist-test-slave-23bp"
server_key: "c7bdd08e9f71c91e13ea817211b106a6"
server_key_iv: "5da9e7bcff7a4d6be83dcbb5e5b4d7c0"
server_key_version: "encryptionkey@0"
I20260928 02:04:47.212975 5414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:47.231020 5414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:47.235387 5414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:47.235698 5414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:47.236707 5414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:47.238013 5414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:47.238083 5414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:47.238246 5414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:47.238302 5414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:47.274379 5414 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:46475
I20260928 02:04:47.274438 5541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:46475 every 8 connection(s)
I20260928 02:04:47.275889 5414 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:04:47.277207 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 5414
I20260928 02:04:47.277421 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:04:47.278154 1754 external_mini_cluster.cc:1428] Setting key ed97faa4b55be33439c0ab583b9b2c8c
Sep 28 02:04:47 dist-test-slave-23bp krb5kdc[5308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.182.130@KRBTEST.COM; defaulting to no policy
I20260928 02:04:47.298882 5544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:47.278402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:55441 (local address 127.1.182.190:39775)
0928 02:04:47.278859 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:47.278866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:47.280365 (+ 1499us) server_negotiation.cc:407] Connection header received
0928 02:04:47.282234 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:47.282242 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:47.282343 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:47.282479 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:47.284035 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:47.285175 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:47.287359 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:47.287774 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:47.287891 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:47.292281 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:47.292285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:47.292289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:47.292320 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:47.295038 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:47.296158 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:47.296162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:47.296165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:47.296280 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:47.296813 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:47.296817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:47.296819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:47.297093 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:47.297200 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:47.298263 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0928 02:04:47.298514 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":248,"threads_started":1}
I20260928 02:04:47.299993 5545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:47.279151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39775 (local address 127.1.182.129:55441)
0928 02:04:47.280192 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:47.280246 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:47.281940 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:47.282709 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:47.282716 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:47.283293 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:47.283861 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:47.283877 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:47.285335 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:47.285338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:47.287232 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:47.287243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:47.288016 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:47.288026 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:47.288353 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:47.289083 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:47.289117 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:47.292045 (+ 2928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:47.295251 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:47.295264 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:47.295287 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:47.296027 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:47.296391 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:47.296409 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:47.296414 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:47.296694 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:47.297225 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:47.297233 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:47.297731 (+ 498us) client_negotiation.cc:769] Sending connection context
0928 02:04:47.298093 (+ 362us) client_negotiation.cc:240] Negotiation successful
0928 02:04:47.298706 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":365,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.130@KRBTEST.COM" created.
I20260928 02:04:47.301780 5542 heartbeater.cc:347] Connected to a master server at 127.1.182.190:39775
I20260928 02:04:47.302202 5542 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:47.303187 5542 heartbeater.cc:511] Master 127.1.182.190:39775 requested a full tablet report, sending...
I20260928 02:04:47.305584 5353 ts_manager.cc:194] Registered new tserver with Master: f55f021aabee481f9c8af32c89d0fc71 (127.1.182.129:46475)
I20260928 02:04:47.307677 5353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:55441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
I20260928 02:04:47.326596 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39775
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:33341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:47.647682 5548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:47.648002 5548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:47.648164 5548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:47.655243 5548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:47.655395 5548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:47.655607 5548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:04:47.665310 5548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:33341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:47.666682 5548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:47.668058 5548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:47.681531 5553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:47.681531 5554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:47.682482 5548 server_base.cc:1034] running on GCE node
W20260928 02:04:47.683290 5556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:47.683941 5548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:47.684985 5548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:47.686229 5548 hybrid_clock.cc:648] HybridClock initialized: now 1790561087686184 us; error 34 us; skew 500 ppm
I20260928 02:04:47.686616 5548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:47 dist-test-slave-23bp krb5kdc[5308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:47.690042 5548 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:04:47.691783 5548 webserver.cc:460] Webserver started at http://127.1.182.130:46387/ using document root <none> and password file <none>
I20260928 02:04:47.692641 5548 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:47.692766 5548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:47.693070 5548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:47.697022 5548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "8429b3c7bee9493fb0f915b6bc0adfa7"
format_stamp: "Formatted at 2026-09-28 02:04:47 on dist-test-slave-23bp"
server_key: "164728a86425327e9806cd6b61247b6f"
server_key_iv: "dd753a5e68742cb6885a7b7bff8022c0"
server_key_version: "encryptionkey@0"
I20260928 02:04:47.697964 5548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "8429b3c7bee9493fb0f915b6bc0adfa7"
format_stamp: "Formatted at 2026-09-28 02:04:47 on dist-test-slave-23bp"
server_key: "164728a86425327e9806cd6b61247b6f"
server_key_iv: "dd753a5e68742cb6885a7b7bff8022c0"
server_key_version: "encryptionkey@0"
I20260928 02:04:47.703334 5548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:04:47.707137 5563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:47.708493 5548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:04:47.708676 5548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "8429b3c7bee9493fb0f915b6bc0adfa7"
format_stamp: "Formatted at 2026-09-28 02:04:47 on dist-test-slave-23bp"
server_key: "164728a86425327e9806cd6b61247b6f"
server_key_iv: "dd753a5e68742cb6885a7b7bff8022c0"
server_key_version: "encryptionkey@0"
I20260928 02:04:47.708884 5548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:47.732537 5548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:47.736920 5548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:47.737228 5548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:47.738251 5548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:47.739709 5548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:47.739815 5548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:47.739960 5548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:47.740032 5548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:47.772447 5548 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:33407
I20260928 02:04:47.772493 5675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:33407 every 8 connection(s)
I20260928 02:04:47.773792 5548 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:04:47.775934 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 5548
I20260928 02:04:47.776151 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:04:47.776604 1754 external_mini_cluster.cc:1428] Setting key 3c6d02824e0f1854b22ce7414b0e5145
Sep 28 02:04:47 dist-test-slave-23bp krb5kdc[5308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.182.131@KRBTEST.COM; defaulting to no policy
I20260928 02:04:47.797324 5544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:47.776796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:50731 (local address 127.1.182.190:39775)
0928 02:04:47.777007 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:47.777013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:47.778318 (+ 1305us) server_negotiation.cc:407] Connection header received
0928 02:04:47.780295 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:47.780300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:47.780398 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:47.780547 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:47.782120 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:47.783289 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:47.785983 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:47.786412 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:47.786531 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:47.790988 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:47.790993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:47.790997 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:47.791028 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:47.793837 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:47.794793 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:47.794797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:47.794799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:47.794915 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:47.795445 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:47.795448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:47.795450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:47.795751 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:47.795867 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:47.796772 (+ 905us) server_negotiation.cc:299] Negotiation successful
0928 02:04:47.796943 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:04:47.798331 5678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:47.777341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39775 (local address 127.1.182.130:50731)
0928 02:04:47.778157 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:47.778212 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:47.780020 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:47.780777 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:47.780785 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:47.781285 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:47.781859 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:47.781907 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:47.783523 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:47.783532 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:47.785809 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:47.785828 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:47.786671 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:47.786681 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:47.787014 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:47.787758 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:47.787792 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:47.790732 (+ 2940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:47.793998 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:47.794010 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:47.794029 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:47.794675 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:47.795031 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:47.795036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:47.795040 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:47.795314 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:47.795896 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:47.795904 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:47.796390 (+ 486us) client_negotiation.cc:769] Sending connection context
0928 02:04:47.796657 (+ 267us) client_negotiation.cc:240] Negotiation successful
0928 02:04:47.797041 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":348,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.131@KRBTEST.COM" created.
I20260928 02:04:47.799853 5676 heartbeater.cc:347] Connected to a master server at 127.1.182.190:39775
I20260928 02:04:47.800215 5676 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:47.801139 5676 heartbeater.cc:511] Master 127.1.182.190:39775 requested a full tablet report, sending...
I20260928 02:04:47.802873 5353 ts_manager.cc:194] Registered new tserver with Master: 8429b3c7bee9493fb0f915b6bc0adfa7 (127.1.182.130:33407)
I20260928 02:04:47.804009 5353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:50731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
I20260928 02:04:47.824383 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39775
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:33341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:48.150544 5681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:48.150821 5681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:48.150977 5681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:48.158159 5681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:48.158310 5681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:48.158514 5681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:04:48.168627 5681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:33341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:48.170012 5681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:48.171471 5681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:48.184649 5687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:48.185144 5686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:48.186901 5689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:48.187942 5681 server_base.cc:1034] running on GCE node
I20260928 02:04:48.188793 5681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:48.190634 5681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:48.191871 5681 hybrid_clock.cc:648] HybridClock initialized: now 1790561088191810 us; error 68 us; skew 500 ppm
I20260928 02:04:48.192225 5681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:48 dist-test-slave-23bp krb5kdc[5308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561088, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:48.195732 5681 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:04:48.197325 5681 webserver.cc:460] Webserver started at http://127.1.182.131:39965/ using document root <none> and password file <none>
I20260928 02:04:48.198050 5681 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:48.198217 5681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:48.198556 5681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:48.201587 5681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "cff119565a614e7fac60a1f499b2e687"
format_stamp: "Formatted at 2026-09-28 02:04:48 on dist-test-slave-23bp"
server_key: "fc7ab8d539e146e7194c5a78b01f8ea0"
server_key_iv: "821b9a05f76e4641ac3e6302a68c4b95"
server_key_version: "encryptionkey@0"
I20260928 02:04:48.202378 5681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "cff119565a614e7fac60a1f499b2e687"
format_stamp: "Formatted at 2026-09-28 02:04:48 on dist-test-slave-23bp"
server_key: "fc7ab8d539e146e7194c5a78b01f8ea0"
server_key_iv: "821b9a05f76e4641ac3e6302a68c4b95"
server_key_version: "encryptionkey@0"
I20260928 02:04:48.208137 5681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 02:04:48.212153 5696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:48.213841 5681 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260928 02:04:48.214030 5681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "cff119565a614e7fac60a1f499b2e687"
format_stamp: "Formatted at 2026-09-28 02:04:48 on dist-test-slave-23bp"
server_key: "fc7ab8d539e146e7194c5a78b01f8ea0"
server_key_iv: "821b9a05f76e4641ac3e6302a68c4b95"
server_key_version: "encryptionkey@0"
I20260928 02:04:48.214288 5681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:48.226706 5681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:48.231313 5681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:48.231612 5681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:48.232575 5681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:48.233999 5681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:48.234107 5681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:48.234261 5681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:48.234336 5681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:48.273917 5681 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:40381
I20260928 02:04:48.273936 5808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:40381 every 8 connection(s)
I20260928 02:04:48.275431 5681 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:04:48.285737 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 5681
I20260928 02:04:48.285951 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:04:48.286537 1754 external_mini_cluster.cc:1428] Setting key d65092ff13cb6ccd336670529a35a48a
Sep 28 02:04:48 dist-test-slave-23bp krb5kdc[5308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561088, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:48.308314 5544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:48.280654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:55769 (local address 127.1.182.190:39775)
0928 02:04:48.280894 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:48.280903 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:48.290172 (+ 9269us) server_negotiation.cc:407] Connection header received
0928 02:04:48.292551 (+ 2379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:48.292556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:48.292649 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:48.292767 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:48.294316 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:48.295178 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:48.296809 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:48.297245 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:48.297384 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:48.301610 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:48.301615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:48.301618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:48.301650 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:48.305061 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:48.305800 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:48.305808 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:48.305813 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:48.305961 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:48.306386 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:48.306393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:48.306396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:48.306711 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:48.306861 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:48.307737 (+ 876us) server_negotiation.cc:299] Negotiation successful
0928 02:04:48.307949 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 02:04:48.309337 5811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:48.279934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39775 (local address 127.1.182.131:55769)
0928 02:04:48.290031 (+ 10097us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:48.290085 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:48.292254 (+ 2169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:48.293023 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:48.293032 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:48.293573 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:48.294153 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:48.294169 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:48.295334 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:48.295339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:48.296660 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:48.296672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:48.297492 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:48.297499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:48.297742 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:48.298452 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:48.298478 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:48.301384 (+ 2906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:48.305203 (+ 3819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:48.305213 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:48.305230 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:48.305675 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:48.306073 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:48.306079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:48.306082 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:48.306273 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:48.306852 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:48.306860 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:48.307325 (+ 465us) client_negotiation.cc:769] Sending connection context
0928 02:04:48.307647 (+ 322us) client_negotiation.cc:240] Negotiation successful
0928 02:04:48.308195 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9785,"thread_start_us":11014,"threads_started":1}
I20260928 02:04:48.310837 5809 heartbeater.cc:347] Connected to a master server at 127.1.182.190:39775
I20260928 02:04:48.310827 5542 heartbeater.cc:503] Master 127.1.182.190:39775 was elected leader, sending a full tablet report...
I20260928 02:04:48.311229 5809 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:48.312150 5809 heartbeater.cc:511] Master 127.1.182.190:39775 requested a full tablet report, sending...
I20260928 02:04:48.313686 5353 ts_manager.cc:194] Registered new tserver with Master: cff119565a614e7fac60a1f499b2e687 (127.1.182.131:40381)
I20260928 02:04:48.314606 5353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:55769
I20260928 02:04:48.316668 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:04:48 dist-test-slave-23bp krb5kdc[5308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:04:48 dist-test-slave-23bp krb5kdc[5308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:04:48.379741 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:48.355905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55924 (local address 127.1.182.129:46475)
0928 02:04:48.359630 (+ 3725us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:48.359638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:48.359666 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:04:48.359726 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:48.359732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:48.359957 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:48.360092 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:48.362370 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:48.363501 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:48.366030 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:48.366645 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:48.366757 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:48.372962 (+ 6205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:48.372979 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:48.372996 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:48.373042 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:48.376405 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:48.377375 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:48.377383 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:48.377385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:48.377528 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:48.378018 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:48.378021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:48.378023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:48.378366 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:48.378716 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:48.379102 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 02:04:48.379371 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3555,"thread_start_us":3399,"threads_started":1}
Sep 28 02:04:48 dist-test-slave-23bp krb5kdc[5308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:48.403198 5544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:48.385103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.1.182.190:39775)
0928 02:04:48.385337 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:48.385346 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:48.385536 (+ 190us) server_negotiation.cc:407] Connection header received
0928 02:04:48.385731 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:48.385739 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:48.385868 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:48.386020 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:48.387385 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:48.388863 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:48.390135 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:48.391111 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:48.391278 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:48.395319 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:48.395329 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:48.395338 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:48.395397 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:48.399879 (+ 4482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:48.400580 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:48.400589 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:48.400595 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:48.400786 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:48.401219 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:48.401227 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:48.401234 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:48.401767 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:48.402085 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:48.402371 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 02:04:48.402652 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260928 02:04:48.403991 5352 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34278
I20260928 02:04:48.405094 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 5414
I20260928 02:04:48.419266 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:48.405972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55928 (local address 127.1.182.129:46475)
0928 02:04:48.406203 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:48.406213 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:48.406654 (+ 441us) server_negotiation.cc:407] Connection header received
0928 02:04:48.406844 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:48.406849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:48.406923 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:48.407029 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:48.408293 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:48.408984 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:48.410180 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:48.410600 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:48.410736 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:48.413222 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:48.413233 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:48.413236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:48.413269 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:48.416456 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:48.417148 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:48.417154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:48.417158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:48.417331 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:48.417724 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:48.417730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:48.417733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:48.418052 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:48.418298 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:48.418658 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 02:04:48.418872 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:04:48.419937 5537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:48.542759 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 5414
I20260928 02:04:48.559863 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 5548
Sep 28 02:04:48 dist-test-slave-23bp krb5kdc[5308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.130@KRBTEST.COM
I20260928 02:04:48.575543 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:48.560898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54988 (local address 127.1.182.130:33407)
0928 02:04:48.561444 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:48.561451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:48.561480 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:04:48.561545 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:48.561552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:48.561764 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:48.561937 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:48.563171 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:48.564167 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:48.565423 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:48.565978 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:48.566116 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:48.570182 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:48.570200 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:48.570215 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:48.570254 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:48.573009 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:48.573721 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:48.573728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:48.573730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:48.573847 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:48.574175 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:48.574179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:48.574181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:48.574519 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:48.574744 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:48.574987 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 02:04:48.575241 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":282,"threads_started":1}
I20260928 02:04:48.576921 5671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:48.660218 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 5548
I20260928 02:04:48.677003 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 5681
Sep 28 02:04:48 dist-test-slave-23bp krb5kdc[5308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.131@KRBTEST.COM
I20260928 02:04:48.693606 5823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:48.677901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56196 (local address 127.1.182.131:40381)
0928 02:04:48.678493 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:48.678500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:48.678535 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:04:48.678594 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:48.678600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:48.678814 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:48.678982 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:48.680293 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:48.681305 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:48.682605 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:48.683244 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:48.683381 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:48.687871 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:48.687882 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:48.687897 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:48.687930 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:48.690686 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:48.691743 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:48.691751 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:48.691753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:48.691869 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:48.692185 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:48.692188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:48.692190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:48.692526 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:48.692744 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:48.693008 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 02:04:48.693237 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":294,"threads_started":1}
I20260928 02:04:48.694932 5804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:48.777755 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 5681
I20260928 02:04:48.793442 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 5321
I20260928 02:04:48.794704 5383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:48.897089 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 5321
2026-09-28T02:04:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3936 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:04:48 dist-test-slave-23bp krb5kdc[5828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:04:48 dist-test-slave-23bp krb5kdc[5828](info): set up 2 sockets
Sep 28 02:04:48 dist-test-slave-23bp krb5kdc[5828](info): commencing operation
krb5kdc: starting...
W20260928 02:04:49.990381 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:04:50 dist-test-slave-23bp krb5kdc[5828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:04:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:04:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:04:50.193979 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:45867
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:45867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:50.515900 5841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:50.516199 5841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:50.516366 5841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:50.523794 5841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:50.523907 5841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:50.524005 5841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:50.524117 5841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:50.524205 5841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:50.534255 5841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:45867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:45867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:50.535645 5841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:50.536983 5841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:50.549078 5846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:50.549047 5847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:50.549057 5849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:50.550100 5841 server_base.cc:1034] running on GCE node
I20260928 02:04:50.550922 5841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:50.552449 5841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:50.553685 5841 hybrid_clock.cc:648] HybridClock initialized: now 1790561090553637 us; error 42 us; skew 500 ppm
I20260928 02:04:50.554045 5841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:50 dist-test-slave-23bp krb5kdc[5828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561090, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:50.560905 5841 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:04:50.562140 5841 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:50.562271 5841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:50.562594 5841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:50.565603 5841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "f815c405b0af44dfb9cc16d75309a5ed"
format_stamp: "Formatted at 2026-09-28 02:04:50 on dist-test-slave-23bp"
server_key: "378ff610b6e1e36f78a76fb9a0340e45"
server_key_iv: "2c2badc172afe66a64f6f3c91183dbc5"
server_key_version: "encryptionkey@0"
I20260928 02:04:50.566373 5841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "f815c405b0af44dfb9cc16d75309a5ed"
format_stamp: "Formatted at 2026-09-28 02:04:50 on dist-test-slave-23bp"
server_key: "378ff610b6e1e36f78a76fb9a0340e45"
server_key_iv: "2c2badc172afe66a64f6f3c91183dbc5"
server_key_version: "encryptionkey@0"
I20260928 02:04:50.571391 5841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 02:04:50.574844 5855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:50.576367 5841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:04:50.576613 5841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "f815c405b0af44dfb9cc16d75309a5ed"
format_stamp: "Formatted at 2026-09-28 02:04:50 on dist-test-slave-23bp"
server_key: "378ff610b6e1e36f78a76fb9a0340e45"
server_key_iv: "2c2badc172afe66a64f6f3c91183dbc5"
server_key_version: "encryptionkey@0"
I20260928 02:04:50.576851 5841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:50.589922 5841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:50.594106 5841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:50.594388 5841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:50.612996 5841 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:45867
I20260928 02:04:50.613021 5906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:45867 every 8 connection(s)
I20260928 02:04:50.613929 5841 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:04:50.617547 5907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:50.622954 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 5841
I20260928 02:04:50.623219 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:04:50.623976 1754 external_mini_cluster.cc:1428] Setting key 1da5dc3a9ccbc945528d45938a1e246f
I20260928 02:04:50.628945 5907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed: Bootstrap starting.
I20260928 02:04:50.632687 5907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:50.633872 5907 log.cc:824] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:50.637969 5907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed: No bootstrap required, opened a new log
Sep 28 02:04:50 dist-test-slave-23bp krb5kdc[5828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:50.647761 5907 raft_consensus.cc:348] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f815c405b0af44dfb9cc16d75309a5ed" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 45867 } }
I20260928 02:04:50.648155 5907 raft_consensus.cc:374] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:50.648262 5907 raft_consensus.cc:729] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f815c405b0af44dfb9cc16d75309a5ed, State: Initialized, Role: FOLLOWER
I20260928 02:04:50.648973 5907 consensus_queue.cc:260] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f815c405b0af44dfb9cc16d75309a5ed" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 45867 } }
I20260928 02:04:50.649235 5907 raft_consensus.cc:388] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:04:50.649374 5907 raft_consensus.cc:482] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:04:50.649556 5907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:50.650554 5910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:50.626189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60974 (local address 127.1.182.190:45867)
0928 02:04:50.627047 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:50.627059 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:50.627204 (+ 145us) server_negotiation.cc:407] Connection header received
0928 02:04:50.628893 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:50.628918 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:50.629370 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:50.629711 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:50.631692 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:50.633173 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:50.635224 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:50.636363 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:50.636540 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:50.642446 (+ 5906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:50.642457 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:50.642477 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:50.642522 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:50.645877 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:50.646595 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:50.646604 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:50.646614 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:50.646791 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:50.647258 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:50.647263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:50.647266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:50.647832 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:50.648133 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:50.648603 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 02:04:50.649079 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":418,"threads_started":1}
I20260928 02:04:50.652308 5907 raft_consensus.cc:504] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f815c405b0af44dfb9cc16d75309a5ed" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 45867 } }
I20260928 02:04:50.652809 5907 leader_election.cc:302] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f815c405b0af44dfb9cc16d75309a5ed; no voters:
I20260928 02:04:50.653469 5907 leader_election.cc:288] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:04:50.653578 5912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:50.654361 5912 raft_consensus.cc:686] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed [term 1 LEADER]: Becoming Leader. State: Replica: f815c405b0af44dfb9cc16d75309a5ed, State: Running, Role: LEADER
I20260928 02:04:50.654932 5912 consensus_queue.cc:237] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f815c405b0af44dfb9cc16d75309a5ed" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 45867 } }
I20260928 02:04:50.655860 5907 sys_catalog.cc:564] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:50.657804 5914 sys_catalog.cc:455] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f815c405b0af44dfb9cc16d75309a5ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f815c405b0af44dfb9cc16d75309a5ed" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 45867 } } }
I20260928 02:04:50.657970 5913 sys_catalog.cc:455] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader f815c405b0af44dfb9cc16d75309a5ed. Latest consensus state: current_term: 1 leader_uuid: "f815c405b0af44dfb9cc16d75309a5ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f815c405b0af44dfb9cc16d75309a5ed" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 45867 } } }
I20260928 02:04:50.658294 5914 sys_catalog.cc:458] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:50.658390 5913 sys_catalog.cc:458] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:50.660966 5919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:50.666041 5919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:50.673789 5919 catalog_manager.cc:1347] Generated new cluster ID: 07806f15032542f18bfdf9a44c4ac7d4
I20260928 02:04:50.673900 5919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:50.682281 5919 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:04:50.683835 5919 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:50.699632 5919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed: Generated new TSK 0
I20260928 02:04:50.700752 5919 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.182.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
I20260928 02:04:50.748804 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:45867
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:51.071147 5933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:51.071457 5933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:51.071619 5933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:51.079200 5933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:51.079312 5933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:51.079505 5933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:04:51.089672 5933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:45867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:51.091012 5933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:51.092382 5933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:51.105649 5939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:51.105638 5938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:51.106271 5941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:51.106592 5933 server_base.cc:1034] running on GCE node
I20260928 02:04:51.107246 5933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:51.108256 5933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:51.109496 5933 hybrid_clock.cc:648] HybridClock initialized: now 1790561091109417 us; error 70 us; skew 500 ppm
I20260928 02:04:51.109824 5933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:51 dist-test-slave-23bp krb5kdc[5828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561091, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:51.113121 5933 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:04:51.114352 5933 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:51.114487 5933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:51.114807 5933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:51.117851 5933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "79faa51804944db589e2a42f38f1b05b"
format_stamp: "Formatted at 2026-09-28 02:04:51 on dist-test-slave-23bp"
server_key: "64d96e7604178a93cbcc51a165e57ff4"
server_key_iv: "33e68211b32483a2b88383f3d433d2e7"
server_key_version: "encryptionkey@0"
I20260928 02:04:51.118683 5933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "79faa51804944db589e2a42f38f1b05b"
format_stamp: "Formatted at 2026-09-28 02:04:51 on dist-test-slave-23bp"
server_key: "64d96e7604178a93cbcc51a165e57ff4"
server_key_iv: "33e68211b32483a2b88383f3d433d2e7"
server_key_version: "encryptionkey@0"
I20260928 02:04:51.123973 5933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:04:51.127477 5947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:51.128799 5933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:04:51.128983 5933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "79faa51804944db589e2a42f38f1b05b"
format_stamp: "Formatted at 2026-09-28 02:04:51 on dist-test-slave-23bp"
server_key: "64d96e7604178a93cbcc51a165e57ff4"
server_key_iv: "33e68211b32483a2b88383f3d433d2e7"
server_key_version: "encryptionkey@0"
I20260928 02:04:51.129220 5933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:51.166380 5933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:51.170650 5933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:51.170957 5933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:51.172024 5933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:51.173461 5933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:51.173595 5933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:51.173727 5933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:51.173815 5933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:51.208627 5933 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:38451
I20260928 02:04:51.208678 6059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:38451 every 8 connection(s)
I20260928 02:04:51.209471 5933 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:04:51.218344 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 5933
I20260928 02:04:51.218577 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:04:51.219473 1754 external_mini_cluster.cc:1428] Setting key 4ef3445c2e3da0b9e1e67b8b4fcf55de
Sep 28 02:04:51 dist-test-slave-23bp krb5kdc[5828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561091, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:51.230489 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:51.212041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:45311 (local address 127.1.182.190:45867)
0928 02:04:51.212508 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:51.212513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:51.213270 (+ 757us) server_negotiation.cc:407] Connection header received
0928 02:04:51.215096 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:51.215103 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:51.215192 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:51.215319 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:51.216756 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:51.217509 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:51.219865 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:51.220410 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:51.220513 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:51.224410 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:51.224415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:51.224419 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:51.224449 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:51.227252 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:51.228037 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:51.228045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:51.228049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:51.228184 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:51.228634 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:51.228642 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:51.228648 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:51.228941 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:51.229107 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:51.229846 (+ 739us) server_negotiation.cc:299] Negotiation successful
0928 02:04:51.230088 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":224,"threads_started":1}
I20260928 02:04:51.231578 6063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:51.212441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:45867 (local address 127.1.182.129:45311)
0928 02:04:51.213107 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:51.213154 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:51.214808 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:51.215551 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:51.215559 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:51.216057 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:51.216593 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:51.216610 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:51.217671 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:51.217681 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:51.219710 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:51.219732 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:51.220652 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:51.220657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:51.220835 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:51.221605 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:51.221634 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:51.224195 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:51.227404 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:51.227423 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:51.227442 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:51.227898 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:51.228305 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:51.228312 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:51.228316 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:51.228513 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:51.229077 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:51.229085 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:51.229443 (+ 358us) client_negotiation.cc:769] Sending connection context
0928 02:04:51.229741 (+ 298us) client_negotiation.cc:240] Negotiation successful
0928 02:04:51.230297 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":269,"threads_started":1}
I20260928 02:04:51.233520 6060 heartbeater.cc:347] Connected to a master server at 127.1.182.190:45867
I20260928 02:04:51.233873 6060 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:51.234537 6060 heartbeater.cc:511] Master 127.1.182.190:45867 requested a full tablet report, sending...
I20260928 02:04:51.237077 5872 ts_manager.cc:194] Registered new tserver with Master: 79faa51804944db589e2a42f38f1b05b (127.1.182.129:38451)
I20260928 02:04:51.239490 5872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:45311
WARNING: no policy specified for kudu/127.1.182.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
I20260928 02:04:51.273454 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:45867
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:51.603150 6066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:51.603449 6066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:51.603626 6066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:51.610702 6066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:51.610837 6066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:51.611032 6066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:04:51.620679 6066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:45867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:51.621930 6066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:51.623266 6066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:51.635715 6071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:51.635751 6072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:51.635788 6074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:51.636694 6066 server_base.cc:1034] running on GCE node
I20260928 02:04:51.637312 6066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:51.638238 6066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:51.639516 6066 hybrid_clock.cc:648] HybridClock initialized: now 1790561091639429 us; error 62 us; skew 500 ppm
I20260928 02:04:51.639884 6066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:51 dist-test-slave-23bp krb5kdc[5828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561091, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:51.642966 6066 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:04:51.644201 6066 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:51.644335 6066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:51.644646 6066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:51.647532 6066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "05573a1a43624e1fb35fdf9688bd0ac1"
format_stamp: "Formatted at 2026-09-28 02:04:51 on dist-test-slave-23bp"
server_key: "487a4f79fee71363d902eba7124bfc33"
server_key_iv: "9de957f4f28f2dd159dea794a1765699"
server_key_version: "encryptionkey@0"
I20260928 02:04:51.648324 6066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "05573a1a43624e1fb35fdf9688bd0ac1"
format_stamp: "Formatted at 2026-09-28 02:04:51 on dist-test-slave-23bp"
server_key: "487a4f79fee71363d902eba7124bfc33"
server_key_iv: "9de957f4f28f2dd159dea794a1765699"
server_key_version: "encryptionkey@0"
I20260928 02:04:51.653146 6066 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:04:51.656703 6080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:51.658097 6066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:04:51.658223 6066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "05573a1a43624e1fb35fdf9688bd0ac1"
format_stamp: "Formatted at 2026-09-28 02:04:51 on dist-test-slave-23bp"
server_key: "487a4f79fee71363d902eba7124bfc33"
server_key_iv: "9de957f4f28f2dd159dea794a1765699"
server_key_version: "encryptionkey@0"
I20260928 02:04:51.658402 6066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:51.676792 6066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:51.680711 6066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:51.680941 6066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:51.681980 6066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:51.683386 6066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:51.683487 6066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:51.683573 6066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:51.683629 6066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:51.716349 6066 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:36397
I20260928 02:04:51.716420 6192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:36397 every 8 connection(s)
I20260928 02:04:51.717209 6066 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:04:51.722604 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 6066
I20260928 02:04:51.722795 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:04:51.723333 1754 external_mini_cluster.cc:1428] Setting key 62506553d4cd3949f328c18d3861d619
Sep 28 02:04:51 dist-test-slave-23bp krb5kdc[5828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561091, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:51.737792 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:51.719724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:35061 (local address 127.1.182.190:45867)
0928 02:04:51.719927 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:51.719933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:51.721133 (+ 1200us) server_negotiation.cc:407] Connection header received
0928 02:04:51.723279 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:51.723283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:51.723351 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:51.723484 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:51.725284 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:51.725976 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:51.727607 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:51.728063 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:51.728200 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:51.731844 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:51.731849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:51.731852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:51.731882 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:51.734736 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:51.735576 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:51.735580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:51.735582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:51.735706 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:51.736134 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:51.736143 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:51.736145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:51.736379 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:51.736537 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:51.737208 (+ 671us) server_negotiation.cc:299] Negotiation successful
0928 02:04:51.737438 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:04:51.738754 6195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:51.720279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:45867 (local address 127.1.182.130:35061)
0928 02:04:51.720978 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:51.721013 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:51.722961 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:51.723710 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:51.723719 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:51.724268 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:51.725113 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:51.725134 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:51.726120 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:51.726125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:51.727467 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:51.727477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:51.728287 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:51.728294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:51.728521 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:51.729209 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:51.729234 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:51.731612 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:51.734877 (+ 3265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:51.734886 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:51.734903 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:51.735406 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:51.735824 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:51.735830 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:51.735834 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:51.736017 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:51.736519 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:51.736528 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:51.736852 (+ 324us) client_negotiation.cc:769] Sending connection context
0928 02:04:51.737120 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 02:04:51.737650 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":266,"threads_started":1}
I20260928 02:04:51.740284 6193 heartbeater.cc:347] Connected to a master server at 127.1.182.190:45867
I20260928 02:04:51.740630 6193 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.182.131@KRBTEST.COM; defaulting to no policy
I20260928 02:04:51.741431 6193 heartbeater.cc:511] Master 127.1.182.190:45867 requested a full tablet report, sending...
I20260928 02:04:51.742955 5872 ts_manager.cc:194] Registered new tserver with Master: 05573a1a43624e1fb35fdf9688bd0ac1 (127.1.182.130:36397)
I20260928 02:04:51.743855 5872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:35061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
I20260928 02:04:51.766258 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:45867
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:04:52.089860 6198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:52.090163 6198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:52.090260 6198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:52.097539 6198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:52.097656 6198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:52.097772 6198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:04:52.108048 6198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:45867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:52.109344 6198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:52.110610 6198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:52.122979 6204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:52.123024 6203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:52.123208 6206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:52.123749 6198 server_base.cc:1034] running on GCE node
I20260928 02:04:52.124306 6198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:52.125245 6198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:52.126492 6198 hybrid_clock.cc:648] HybridClock initialized: now 1790561092126428 us; error 67 us; skew 500 ppm
I20260928 02:04:52.126822 6198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:04:52 dist-test-slave-23bp krb5kdc[5828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561092, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:04:52.130095 6198 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:04:52.131286 6198 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:52.131418 6198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:52.131736 6198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:52.134690 6198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "b6808a75c9984523802e0486ae8411ef"
format_stamp: "Formatted at 2026-09-28 02:04:52 on dist-test-slave-23bp"
server_key: "09f18ba7432e761dec5ad1caee766d0f"
server_key_iv: "4f0385ccd219aa4d2d9891cbddd14c3d"
server_key_version: "encryptionkey@0"
I20260928 02:04:52.135540 6198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "b6808a75c9984523802e0486ae8411ef"
format_stamp: "Formatted at 2026-09-28 02:04:52 on dist-test-slave-23bp"
server_key: "09f18ba7432e761dec5ad1caee766d0f"
server_key_iv: "4f0385ccd219aa4d2d9891cbddd14c3d"
server_key_version: "encryptionkey@0"
I20260928 02:04:52.140522 6198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 02:04:52.143887 6212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:52.145188 6198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:04:52.145356 6198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "b6808a75c9984523802e0486ae8411ef"
format_stamp: "Formatted at 2026-09-28 02:04:52 on dist-test-slave-23bp"
server_key: "09f18ba7432e761dec5ad1caee766d0f"
server_key_iv: "4f0385ccd219aa4d2d9891cbddd14c3d"
server_key_version: "encryptionkey@0"
I20260928 02:04:52.145570 6198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:52.162926 6198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:52.166788 6198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:52.167096 6198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:52.168044 6198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:52.169353 6198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:52.169474 6198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:52.169616 6198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:52.169713 6198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:52.202562 6198 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:46737
I20260928 02:04:52.202646 6324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:46737 every 8 connection(s)
I20260928 02:04:52.203438 6198 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:04:52.205264 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 6198
I20260928 02:04:52.205459 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:04:52.205940 1754 external_mini_cluster.cc:1428] Setting key 23dba18d69045c37c670fbe0c45c4725
Sep 28 02:04:52 dist-test-slave-23bp krb5kdc[5828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561092, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:04:52.226372 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:52.206216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:59309 (local address 127.1.182.190:45867)
0928 02:04:52.206407 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:52.206413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:52.207747 (+ 1334us) server_negotiation.cc:407] Connection header received
0928 02:04:52.212935 (+ 5188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:52.212938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:52.213025 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:52.213116 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:52.214491 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.215278 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.216755 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.217182 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.217295 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:52.220854 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:52.220858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:52.220862 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:52.220895 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:52.223472 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:52.224166 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:52.224173 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:52.224177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:52.224304 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:52.224718 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:52.224733 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:52.224736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:52.224994 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:52.225154 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:52.225814 (+ 660us) server_negotiation.cc:299] Negotiation successful
0928 02:04:52.226023 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:04:52.227365 6327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:52.206756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:45867 (local address 127.1.182.131:59309)
0928 02:04:52.207569 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:52.207612 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:52.212688 (+ 5076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:52.213305 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:52.213313 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:52.213797 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:52.214340 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:52.214360 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.215404 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.215408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:52.216605 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:52.216616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.217363 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.217367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:52.217541 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:52.218208 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:04:52.218229 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:52.220641 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:52.223604 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:52.223613 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:52.223629 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:52.224043 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:52.224416 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:04:52.224424 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:04:52.224428 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:04:52.224610 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:04:52.225127 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:52.225134 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:52.225467 (+ 333us) client_negotiation.cc:769] Sending connection context
0928 02:04:52.225729 (+ 262us) client_negotiation.cc:240] Negotiation successful
0928 02:04:52.226241 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":357,"threads_started":1}
I20260928 02:04:52.228710 6325 heartbeater.cc:347] Connected to a master server at 127.1.182.190:45867
I20260928 02:04:52.228971 6325 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:52.229575 6325 heartbeater.cc:511] Master 127.1.182.190:45867 requested a full tablet report, sending...
I20260928 02:04:52.230965 5872 ts_manager.cc:194] Registered new tserver with Master: b6808a75c9984523802e0486ae8411ef (127.1.182.131:46737)
I20260928 02:04:52.231844 5872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:59309
I20260928 02:04:52.233623 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:04:52.242821 6060 heartbeater.cc:503] Master 127.1.182.190:45867 was elected leader, sending a full tablet report...
I20260928 02:04:52.251012 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:52.238008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.1.182.190:45867)
0928 02:04:52.238165 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:52.238169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:52.238845 (+ 676us) server_negotiation.cc:407] Connection header received
0928 02:04:52.239053 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:52.239058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:52.239150 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:52.239258 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:52.240721 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.241436 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.242711 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.243307 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.243431 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:52.245971 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:52.245976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:52.245979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:52.246007 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:52.248455 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:52.249098 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:52.249102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:52.249104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:52.249218 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:52.249628 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:52.249633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:52.249638 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:52.250047 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:52.250214 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:52.250520 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 02:04:52.250685 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:04:52.254971 5871 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:04:52.262180 5871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:04:52.308393 6338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:52.273698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:46737 (local address 127.0.0.1:50330)
0928 02:04:52.299837 (+ 26139us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:52.299864 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:52.300052 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:52.301404 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:52.301411 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:52.301451 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:52.301863 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:52.301876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.303828 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.303836 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:52.306001 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:52.306012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.307333 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.307339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:52.307642 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:52.307680 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:04:52.307820 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 02:04:52.308012 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25956,"thread_start_us":27491,"threads_started":1}
I20260928 02:04:52.309160 6341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:52.274409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50330 (local address 127.1.182.131:46737)
0928 02:04:52.299478 (+ 25069us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:52.299491 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:52.300218 (+ 727us) server_negotiation.cc:407] Connection header received
0928 02:04:52.300358 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:52.300370 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:52.300710 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:52.300963 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:52.302134 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.303658 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.306148 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.307152 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.307298 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:52.307545 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:52.307864 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 02:04:52.308253 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24837,"thread_start_us":24604,"threads_started":1}
I20260928 02:04:52.310129 6337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:52.273991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:36397 (local address 127.0.0.1:33534)
0928 02:04:52.301182 (+ 27191us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:52.301200 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:52.301424 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:52.302460 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:52.302466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:52.302493 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:52.302844 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:52.302855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.304689 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.304696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:52.307146 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:52.307165 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.309279 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.309285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:52.309455 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:52.309485 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 02:04:52.309595 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:04:52.309740 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27022,"mutex_wait_us":62,"thread_start_us":25561,"threads_started":1}
I20260928 02:04:52.310145 6342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:52.274648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33534 (local address 127.1.182.130:36397)
0928 02:04:52.297153 (+ 22505us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:52.297168 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:52.301315 (+ 4147us) server_negotiation.cc:407] Connection header received
0928 02:04:52.301985 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:52.301995 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:52.302250 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:52.302451 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:52.303034 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.304512 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.307330 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.308806 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.308920 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:52.309104 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:52.309671 (+ 567us) server_negotiation.cc:299] Negotiation successful
0928 02:04:52.309828 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22244,"thread_start_us":22522,"threads_started":1}
I20260928 02:04:52.310545 6340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:52.273600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:38451 (local address 127.0.0.1:49318)
0928 02:04:52.298157 (+ 24557us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:52.298220 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:52.298484 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:52.301842 (+ 3358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:52.301847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:52.301870 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:52.302301 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:52.302312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.304357 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.304362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:52.307175 (+ 2813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:52.307184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.309734 (+ 2550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.309740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:52.309910 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:52.309941 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 02:04:52.310028 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:04:52.310191 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24195,"mutex_wait_us":328,"thread_start_us":27278,"threads_started":1}
I20260928 02:04:52.311254 6339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:52.273739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49318 (local address 127.1.182.129:38451)
0928 02:04:52.300845 (+ 27106us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:52.300855 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:52.300888 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:04:52.300994 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:52.301004 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:52.301270 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:52.301572 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:52.302656 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.304165 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.307330 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.309253 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.309373 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:52.309607 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:52.310845 (+ 1238us) server_negotiation.cc:299] Negotiation successful
0928 02:04:52.310956 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26939,"thread_start_us":26789,"threads_started":1}
I20260928 02:04:52.313496 6260 tablet_service.cc:1467] Processing CreateTablet for tablet 920987893a9844f1a62919f7442be95e (DEFAULT_TABLE table=test-table [id=da65f8979a714e3091d8ca48124e8786]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:52.313922 5995 tablet_service.cc:1467] Processing CreateTablet for tablet 920987893a9844f1a62919f7442be95e (DEFAULT_TABLE table=test-table [id=da65f8979a714e3091d8ca48124e8786]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:52.313947 6128 tablet_service.cc:1467] Processing CreateTablet for tablet 920987893a9844f1a62919f7442be95e (DEFAULT_TABLE table=test-table [id=da65f8979a714e3091d8ca48124e8786]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:52.314955 6260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920987893a9844f1a62919f7442be95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:52.315050 5995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920987893a9844f1a62919f7442be95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:52.315196 6128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920987893a9844f1a62919f7442be95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:52.326380 6343 tablet_bootstrap.cc:492] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b: Bootstrap starting.
I20260928 02:04:52.329088 6344 tablet_bootstrap.cc:492] T 920987893a9844f1a62919f7442be95e P b6808a75c9984523802e0486ae8411ef: Bootstrap starting.
I20260928 02:04:52.329712 6343 tablet_bootstrap.cc:654] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:52.330646 6345 tablet_bootstrap.cc:492] T 920987893a9844f1a62919f7442be95e P 05573a1a43624e1fb35fdf9688bd0ac1: Bootstrap starting.
I20260928 02:04:52.331652 6343 log.cc:824] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:52.332670 6344 tablet_bootstrap.cc:654] T 920987893a9844f1a62919f7442be95e P b6808a75c9984523802e0486ae8411ef: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:52.333820 6344 log.cc:824] T 920987893a9844f1a62919f7442be95e P b6808a75c9984523802e0486ae8411ef: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:52.333958 6345 tablet_bootstrap.cc:654] T 920987893a9844f1a62919f7442be95e P 05573a1a43624e1fb35fdf9688bd0ac1: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:52.334985 6343 tablet_bootstrap.cc:492] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b: No bootstrap required, opened a new log
I20260928 02:04:52.335213 6345 log.cc:824] T 920987893a9844f1a62919f7442be95e P 05573a1a43624e1fb35fdf9688bd0ac1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:52.335278 6343 ts_tablet_manager.cc:1397] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 02:04:52.336859 6344 tablet_bootstrap.cc:492] T 920987893a9844f1a62919f7442be95e P b6808a75c9984523802e0486ae8411ef: No bootstrap required, opened a new log
I20260928 02:04:52.337095 6344 ts_tablet_manager.cc:1397] T 920987893a9844f1a62919f7442be95e P b6808a75c9984523802e0486ae8411ef: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 02:04:52.338156 6345 tablet_bootstrap.cc:492] T 920987893a9844f1a62919f7442be95e P 05573a1a43624e1fb35fdf9688bd0ac1: No bootstrap required, opened a new log
I20260928 02:04:52.338505 6345 ts_tablet_manager.cc:1397] T 920987893a9844f1a62919f7442be95e P 05573a1a43624e1fb35fdf9688bd0ac1: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 02:04:52.345098 6345 raft_consensus.cc:348] T 920987893a9844f1a62919f7442be95e P 05573a1a43624e1fb35fdf9688bd0ac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79faa51804944db589e2a42f38f1b05b" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 38451 } } peers { permanent_uuid: "b6808a75c9984523802e0486ae8411ef" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46737 } } peers { permanent_uuid: "05573a1a43624e1fb35fdf9688bd0ac1" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 36397 } }
I20260928 02:04:52.345466 6345 raft_consensus.cc:374] T 920987893a9844f1a62919f7442be95e P 05573a1a43624e1fb35fdf9688bd0ac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:52.345568 6345 raft_consensus.cc:729] T 920987893a9844f1a62919f7442be95e P 05573a1a43624e1fb35fdf9688bd0ac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05573a1a43624e1fb35fdf9688bd0ac1, State: Initialized, Role: FOLLOWER
I20260928 02:04:52.345712 6343 raft_consensus.cc:348] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79faa51804944db589e2a42f38f1b05b" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 38451 } } peers { permanent_uuid: "b6808a75c9984523802e0486ae8411ef" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46737 } } peers { permanent_uuid: "05573a1a43624e1fb35fdf9688bd0ac1" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 36397 } }
I20260928 02:04:52.346285 6343 raft_consensus.cc:374] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:52.346192 6345 consensus_queue.cc:260] T 920987893a9844f1a62919f7442be95e P 05573a1a43624e1fb35fdf9688bd0ac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79faa51804944db589e2a42f38f1b05b" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 38451 } } peers { permanent_uuid: "b6808a75c9984523802e0486ae8411ef" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46737 } } peers { permanent_uuid: "05573a1a43624e1fb35fdf9688bd0ac1" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 36397 } }
I20260928 02:04:52.346413 6343 raft_consensus.cc:729] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79faa51804944db589e2a42f38f1b05b, State: Initialized, Role: FOLLOWER
I20260928 02:04:52.347081 6343 consensus_queue.cc:260] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79faa51804944db589e2a42f38f1b05b" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 38451 } } peers { permanent_uuid: "b6808a75c9984523802e0486ae8411ef" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46737 } } peers { permanent_uuid: "05573a1a43624e1fb35fdf9688bd0ac1" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 36397 } }
I20260928 02:04:52.347875 6193 heartbeater.cc:503] Master 127.1.182.190:45867 was elected leader, sending a full tablet report...
I20260928 02:04:52.347747 6344 raft_consensus.cc:348] T 920987893a9844f1a62919f7442be95e P b6808a75c9984523802e0486ae8411ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79faa51804944db589e2a42f38f1b05b" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 38451 } } peers { permanent_uuid: "b6808a75c9984523802e0486ae8411ef" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46737 } } peers { permanent_uuid: "05573a1a43624e1fb35fdf9688bd0ac1" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 36397 } }
I20260928 02:04:52.348187 6344 raft_consensus.cc:374] T 920987893a9844f1a62919f7442be95e P b6808a75c9984523802e0486ae8411ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:52.348317 6344 raft_consensus.cc:729] T 920987893a9844f1a62919f7442be95e P b6808a75c9984523802e0486ae8411ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6808a75c9984523802e0486ae8411ef, State: Initialized, Role: FOLLOWER
I20260928 02:04:52.348969 6344 consensus_queue.cc:260] T 920987893a9844f1a62919f7442be95e P b6808a75c9984523802e0486ae8411ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79faa51804944db589e2a42f38f1b05b" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 38451 } } peers { permanent_uuid: "b6808a75c9984523802e0486ae8411ef" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46737 } } peers { permanent_uuid: "05573a1a43624e1fb35fdf9688bd0ac1" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 36397 } }
I20260928 02:04:52.350402 6345 ts_tablet_manager.cc:1428] T 920987893a9844f1a62919f7442be95e P 05573a1a43624e1fb35fdf9688bd0ac1: Time spent starting tablet: real 0.012s user 0.004s sys 0.006s
I20260928 02:04:52.350651 6343 ts_tablet_manager.cc:1428] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260928 02:04:52.351431 6325 heartbeater.cc:503] Master 127.1.182.190:45867 was elected leader, sending a full tablet report...
I20260928 02:04:52.352139 6344 ts_tablet_manager.cc:1428] T 920987893a9844f1a62919f7442be95e P b6808a75c9984523802e0486ae8411ef: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260928 02:04:52.364955 6350 raft_consensus.cc:482] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:52.365491 6350 raft_consensus.cc:504] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79faa51804944db589e2a42f38f1b05b" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 38451 } } peers { permanent_uuid: "b6808a75c9984523802e0486ae8411ef" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46737 } } peers { permanent_uuid: "05573a1a43624e1fb35fdf9688bd0ac1" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 36397 } }
I20260928 02:04:52.368206 6350 leader_election.cc:288] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6808a75c9984523802e0486ae8411ef (127.1.182.131:46737), 05573a1a43624e1fb35fdf9688bd0ac1 (127.1.182.130:36397)
I20260928 02:04:52.377630 6341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:52.368761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:55841 (local address 127.1.182.131:46737)
0928 02:04:52.368962 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:52.368972 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:52.369471 (+ 499us) server_negotiation.cc:407] Connection header received
0928 02:04:52.369745 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:52.369751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:52.369820 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:52.369923 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:52.370584 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.372031 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.375448 (+ 3417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.376356 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.376457 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:52.376506 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:52.377250 (+ 744us) server_negotiation.cc:299] Negotiation successful
0928 02:04:52.377370 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:04:52.377673 6353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:52.368375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:46737 (local address 127.1.182.129:55841)
0928 02:04:52.369349 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:52.369374 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:52.369594 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:52.369960 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:52.369967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:52.370024 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:52.370400 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:52.370414 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.372262 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.372270 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:52.375227 (+ 2957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:52.375247 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.376758 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.376769 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:52.377071 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:52.377112 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 02:04:52.377228 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 02:04:52.377356 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":6903,"threads_started":1}
I20260928 02:04:52.378904 6280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920987893a9844f1a62919f7442be95e" candidate_uuid: "79faa51804944db589e2a42f38f1b05b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6808a75c9984523802e0486ae8411ef" is_pre_election: true
I20260928 02:04:52.379528 6280 raft_consensus.cc:2454] T 920987893a9844f1a62919f7442be95e P b6808a75c9984523802e0486ae8411ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79faa51804944db589e2a42f38f1b05b in term 0.
I20260928 02:04:52.380173 5951 leader_election.cc:302] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79faa51804944db589e2a42f38f1b05b, b6808a75c9984523802e0486ae8411ef; no voters:
I20260928 02:04:52.380564 6350 raft_consensus.cc:2781] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:04:52.380721 6350 raft_consensus.cc:482] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:04:52.380824 6350 raft_consensus.cc:3037] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:52.383834 6342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:52.368660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:48537 (local address 127.1.182.130:36397)
0928 02:04:52.368850 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:52.368856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:52.375990 (+ 7134us) server_negotiation.cc:407] Connection header received
0928 02:04:52.376260 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:52.376269 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:52.376357 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:52.376422 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:04:52.377359 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.378332 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.381237 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.382543 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.382674 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:52.382751 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:52.383412 (+ 661us) server_negotiation.cc:299] Negotiation successful
0928 02:04:52.383562 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:04:52.384387 6350 raft_consensus.cc:504] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79faa51804944db589e2a42f38f1b05b" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 38451 } } peers { permanent_uuid: "b6808a75c9984523802e0486ae8411ef" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46737 } } peers { permanent_uuid: "05573a1a43624e1fb35fdf9688bd0ac1" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 36397 } }
I20260928 02:04:52.384814 6352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:52.368389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:36397 (local address 127.1.182.129:48537)
0928 02:04:52.375837 (+ 7448us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:52.375861 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:52.376101 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:52.376634 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:52.376639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:52.376667 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:04:52.377168 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:52.377181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.378521 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.378526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:52.381039 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:52.381058 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.382986 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.382994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:52.383216 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:52.383253 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:04:52.384095 (+ 842us) client_negotiation.cc:240] Negotiation successful
0928 02:04:52.384273 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7261,"mutex_wait_us":280,"thread_start_us":294,"threads_started":1}
I20260928 02:04:52.385178 6350 leader_election.cc:288] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b [CANDIDATE]: Term 1 election: Requested vote from peers b6808a75c9984523802e0486ae8411ef (127.1.182.131:46737), 05573a1a43624e1fb35fdf9688bd0ac1 (127.1.182.130:36397)
I20260928 02:04:52.385483 6280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920987893a9844f1a62919f7442be95e" candidate_uuid: "79faa51804944db589e2a42f38f1b05b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6808a75c9984523802e0486ae8411ef"
I20260928 02:04:52.385671 6280 raft_consensus.cc:3037] T 920987893a9844f1a62919f7442be95e P b6808a75c9984523802e0486ae8411ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:52.385792 6148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920987893a9844f1a62919f7442be95e" candidate_uuid: "79faa51804944db589e2a42f38f1b05b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05573a1a43624e1fb35fdf9688bd0ac1" is_pre_election: true
I20260928 02:04:52.386313 6148 raft_consensus.cc:2454] T 920987893a9844f1a62919f7442be95e P 05573a1a43624e1fb35fdf9688bd0ac1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79faa51804944db589e2a42f38f1b05b in term 0.
I20260928 02:04:52.387336 6148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920987893a9844f1a62919f7442be95e" candidate_uuid: "79faa51804944db589e2a42f38f1b05b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05573a1a43624e1fb35fdf9688bd0ac1"
I20260928 02:04:52.387562 6148 raft_consensus.cc:3037] T 920987893a9844f1a62919f7442be95e P 05573a1a43624e1fb35fdf9688bd0ac1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:52.388388 6280 raft_consensus.cc:2454] T 920987893a9844f1a62919f7442be95e P b6808a75c9984523802e0486ae8411ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79faa51804944db589e2a42f38f1b05b in term 1.
I20260928 02:04:52.388772 5951 leader_election.cc:302] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79faa51804944db589e2a42f38f1b05b, b6808a75c9984523802e0486ae8411ef; no voters:
I20260928 02:04:52.389032 6350 raft_consensus.cc:2781] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:52.389518 6350 raft_consensus.cc:686] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b [term 1 LEADER]: Becoming Leader. State: Replica: 79faa51804944db589e2a42f38f1b05b, State: Running, Role: LEADER
I20260928 02:04:52.390014 6148 raft_consensus.cc:2454] T 920987893a9844f1a62919f7442be95e P 05573a1a43624e1fb35fdf9688bd0ac1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79faa51804944db589e2a42f38f1b05b in term 1.
I20260928 02:04:52.390048 6350 consensus_queue.cc:237] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79faa51804944db589e2a42f38f1b05b" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 38451 } } peers { permanent_uuid: "b6808a75c9984523802e0486ae8411ef" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46737 } } peers { permanent_uuid: "05573a1a43624e1fb35fdf9688bd0ac1" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 36397 } }
I20260928 02:04:52.395655 5870 catalog_manager.cc:5495] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b reported cstate change: term changed from 0 to 1, leader changed from <none> to 79faa51804944db589e2a42f38f1b05b (127.1.182.129). New cstate: current_term: 1 leader_uuid: "79faa51804944db589e2a42f38f1b05b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79faa51804944db589e2a42f38f1b05b" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 38451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6808a75c9984523802e0486ae8411ef" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05573a1a43624e1fb35fdf9688bd0ac1" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 36397 } health_report { overall_health: UNKNOWN } } }
I20260928 02:04:52.418510 6339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:52.412978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49324 (local address 127.1.182.129:38451)
0928 02:04:52.413199 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:52.413206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:52.413230 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:04:52.413395 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:52.413402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:52.413534 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:52.413716 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:04:52.413993 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.415230 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.416595 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.416986 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.417085 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:52.417508 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:04:52.418008 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:04:52.418118 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:52.418200 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 02:04:52.418261 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:04:52.426508 6148 raft_consensus.cc:1261] T 920987893a9844f1a62919f7442be95e P 05573a1a43624e1fb35fdf9688bd0ac1 [term 1 FOLLOWER]: Refusing update from remote peer 79faa51804944db589e2a42f38f1b05b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:04:52.426550 6280 raft_consensus.cc:1261] T 920987893a9844f1a62919f7442be95e P b6808a75c9984523802e0486ae8411ef [term 1 FOLLOWER]: Refusing update from remote peer 79faa51804944db589e2a42f38f1b05b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:04:52.427516 6350 consensus_queue.cc:1035] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05573a1a43624e1fb35fdf9688bd0ac1" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:52.427794 6354 consensus_queue.cc:1035] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6808a75c9984523802e0486ae8411ef" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:52.440704 6357 mvcc.cc:205] Tried to move back new op lower bound from 7334138234569830400 to 7334138234434269184. Current Snapshot: MvccSnapshot[applied={T|T < 7334138234569830400}]
I20260928 02:04:52.467414 5871 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:04:52.467973 5871 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:04:52.471553 5871 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f815c405b0af44dfb9cc16d75309a5ed: Sending DeleteTablet for 3 replicas of tablet 920987893a9844f1a62919f7442be95e
I20260928 02:04:52.472684 5995 tablet_service.cc:1514] Processing DeleteTablet for tablet 920987893a9844f1a62919f7442be95e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:52 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:49318
I20260928 02:04:52.472679 6260 tablet_service.cc:1514] Processing DeleteTablet for tablet 920987893a9844f1a62919f7442be95e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:52 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:50330
I20260928 02:04:52.473433 6128 tablet_service.cc:1514] Processing DeleteTablet for tablet 920987893a9844f1a62919f7442be95e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:52 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:33534
I20260928 02:04:52.473691 6369 tablet_replica.cc:330] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b: stopping tablet replica
I20260928 02:04:52.474160 6369 raft_consensus.cc:2229] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:04:52.474879 6369 raft_consensus.cc:2258] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:52.475903 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 5933
I20260928 02:04:52.483029 6369 ts_tablet_manager.cc:1905] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 02:04:52 dist-test-slave-23bp krb5kdc[5828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:04:52.491047 6370 tablet_replica.cc:330] T 920987893a9844f1a62919f7442be95e P 05573a1a43624e1fb35fdf9688bd0ac1: stopping tablet replica
I20260928 02:04:52.491585 6370 raft_consensus.cc:2229] T 920987893a9844f1a62919f7442be95e P 05573a1a43624e1fb35fdf9688bd0ac1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:04:52.492149 6370 raft_consensus.cc:2258] T 920987893a9844f1a62919f7442be95e P 05573a1a43624e1fb35fdf9688bd0ac1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:52.497757 6370 ts_tablet_manager.cc:1905] T 920987893a9844f1a62919f7442be95e P 05573a1a43624e1fb35fdf9688bd0ac1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:52.498795 6369 ts_tablet_manager.cc:1918] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:04:52.498983 6369 log.cc:1196] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-0/wal/wals/920987893a9844f1a62919f7442be95e
I20260928 02:04:52.500437 6369 ts_tablet_manager.cc:1939] T 920987893a9844f1a62919f7442be95e P 79faa51804944db589e2a42f38f1b05b: Deleting consensus metadata
I20260928 02:04:52.500463 6339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:52.477078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.1.182.129:38451)
0928 02:04:52.477249 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:52.477258 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:52.479556 (+ 2298us) server_negotiation.cc:407] Connection header received
0928 02:04:52.479815 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:52.479821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:52.479911 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:52.479984 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:52.481901 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.483225 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.485322 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.486016 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.486133 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:52.491683 (+ 5550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:52.491698 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:52.491718 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:52.491765 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:52.496251 (+ 4486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:52.497145 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:52.497153 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:52.497159 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:52.497365 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:52.497836 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:52.497841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:52.497844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:52.498254 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:52.498517 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:52.499179 (+ 662us) server_negotiation.cc:299] Negotiation successful
0928 02:04:52.499423 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:04:52.501149 6055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:52.500133 6368 tablet_replica.cc:330] T 920987893a9844f1a62919f7442be95e P b6808a75c9984523802e0486ae8411ef: stopping tablet replica
I20260928 02:04:52.501734 6368 raft_consensus.cc:2229] T 920987893a9844f1a62919f7442be95e P b6808a75c9984523802e0486ae8411ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:04:52.502249 6368 raft_consensus.cc:2258] T 920987893a9844f1a62919f7442be95e P b6808a75c9984523802e0486ae8411ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:52.504360 6368 ts_tablet_manager.cc:1905] T 920987893a9844f1a62919f7442be95e P b6808a75c9984523802e0486ae8411ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:52.508332 6370 ts_tablet_manager.cc:1918] T 920987893a9844f1a62919f7442be95e P 05573a1a43624e1fb35fdf9688bd0ac1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:04:52.508504 6370 log.cc:1196] T 920987893a9844f1a62919f7442be95e P 05573a1a43624e1fb35fdf9688bd0ac1: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/920987893a9844f1a62919f7442be95e
I20260928 02:04:52.509032 6370 ts_tablet_manager.cc:1939] T 920987893a9844f1a62919f7442be95e P 05573a1a43624e1fb35fdf9688bd0ac1: Deleting consensus metadata
I20260928 02:04:52.510959 5856 catalog_manager.cc:4841] TS 05573a1a43624e1fb35fdf9688bd0ac1 (127.1.182.130:36397): tablet 920987893a9844f1a62919f7442be95e (table test-table [id=da65f8979a714e3091d8ca48124e8786]) successfully deleted
I20260928 02:04:52.511296 6368 ts_tablet_manager.cc:1918] T 920987893a9844f1a62919f7442be95e P b6808a75c9984523802e0486ae8411ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:04:52.511467 6368 log.cc:1196] T 920987893a9844f1a62919f7442be95e P b6808a75c9984523802e0486ae8411ef: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/920987893a9844f1a62919f7442be95e
I20260928 02:04:52.511905 6368 ts_tablet_manager.cc:1939] T 920987893a9844f1a62919f7442be95e P b6808a75c9984523802e0486ae8411ef: Deleting consensus metadata
I20260928 02:04:52.513314 5859 catalog_manager.cc:4841] TS b6808a75c9984523802e0486ae8411ef (127.1.182.131:46737): tablet 920987893a9844f1a62919f7442be95e (table test-table [id=da65f8979a714e3091d8ca48124e8786]) successfully deleted
I20260928 02:04:52.599563 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 5933
I20260928 02:04:52.600019 5858 catalog_manager.cc:4841] TS 79faa51804944db589e2a42f38f1b05b (127.1.182.129:38451): tablet 920987893a9844f1a62919f7442be95e (table test-table [id=da65f8979a714e3091d8ca48124e8786]) successfully deleted
I20260928 02:04:52.617775 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 6066
Sep 28 02:04:52 dist-test-slave-23bp krb5kdc[5828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.130@KRBTEST.COM
I20260928 02:04:52.634276 6342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:52.618582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33538 (local address 127.1.182.130:36397)
0928 02:04:52.618780 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:52.618785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:52.618802 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:04:52.618978 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:52.618982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:52.619048 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:52.619178 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:52.620705 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.621493 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.623301 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.623777 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.623877 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:52.628816 (+ 4939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:52.628832 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:52.628848 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:52.628891 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:52.631938 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:52.632546 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:52.632561 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:52.632563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:52.632682 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:52.632970 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:52.632973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:52.632975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:52.633324 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:52.633526 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:52.633769 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 02:04:52.633924 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 02:04:52.634928 6188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:52.722776 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 6066
I20260928 02:04:52.739194 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 6198
Sep 28 02:04:52 dist-test-slave-23bp krb5kdc[5828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.131@KRBTEST.COM
I20260928 02:04:52.754139 6341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:52.740165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50344 (local address 127.1.182.131:46737)
0928 02:04:52.740368 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:52.740374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:52.740466 (+ 92us) server_negotiation.cc:407] Connection header received
0928 02:04:52.740674 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:52.740680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:52.740768 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:52.740877 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:52.742069 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.742790 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.744196 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:52.744606 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:52.744749 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:52.749175 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:52.749191 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:52.749208 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:04:52.749245 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:52.751957 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:52.752516 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:52.752523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:52.752525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:52.752652 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:04:52.752949 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:04:52.752952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:04:52.752954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:04:52.753287 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:52.753484 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:52.753736 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 02:04:52.753891 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:04:52.754768 6320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:52.843868 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 6198
I20260928 02:04:52.860004 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 5841
I20260928 02:04:52.861159 5902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:52.968086 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 5841
2026-09-28T02:04:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4069 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T02:04:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:04:53Z Disabled control of system clock
I20260928 02:04:53.033272 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:40603
--webserver_interface=127.1.182.190
--webserver_port=0
--builtin_ntp_servers=127.1.182.148:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:40603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:04:53.357833 6379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:53.358109 6379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:53.358191 6379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:53.365311 6379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:53.365422 6379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:53.365461 6379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:53.365518 6379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:53.365554 6379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:04:53.374468 6379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:40603
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:40603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:53.375710 6379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:53.377400 6379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:53.389130 6384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:53.389178 6387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:53.389202 6385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:53.389695 6379 server_base.cc:1034] running on GCE node
I20260928 02:04:53.390458 6379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:53.391885 6379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:53.393124 6379 hybrid_clock.cc:648] HybridClock initialized: now 1790561093393075 us; error 46 us; skew 500 ppm
I20260928 02:04:53.393461 6379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:53.395828 6379 webserver.cc:460] Webserver started at http://127.1.182.190:38765/ using document root <none> and password file <none>
I20260928 02:04:53.396509 6379 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:53.396639 6379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:53.396957 6379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:53.399837 6379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "ec66c9bb391848df8b98e4f967380d8f"
format_stamp: "Formatted at 2026-09-28 02:04:53 on dist-test-slave-23bp"
server_key: "f82295589bae4f9e8a8386ade7e0aade"
server_key_iv: "3c669ddb355b1eadccb74618d1f07667"
server_key_version: "encryptionkey@0"
I20260928 02:04:53.400597 6379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "ec66c9bb391848df8b98e4f967380d8f"
format_stamp: "Formatted at 2026-09-28 02:04:53 on dist-test-slave-23bp"
server_key: "f82295589bae4f9e8a8386ade7e0aade"
server_key_iv: "3c669ddb355b1eadccb74618d1f07667"
server_key_version: "encryptionkey@0"
I20260928 02:04:53.405546 6379 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:04:53.408903 6393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:53.410264 6379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:04:53.410436 6379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "ec66c9bb391848df8b98e4f967380d8f"
format_stamp: "Formatted at 2026-09-28 02:04:53 on dist-test-slave-23bp"
server_key: "f82295589bae4f9e8a8386ade7e0aade"
server_key_iv: "3c669ddb355b1eadccb74618d1f07667"
server_key_version: "encryptionkey@0"
I20260928 02:04:53.410646 6379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:53.416320 6379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:53.417059 6379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:53.417347 6379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:53.435343 6379 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:40603
I20260928 02:04:53.435374 6444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:40603 every 8 connection(s)
I20260928 02:04:53.436642 6379 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:04:53.440104 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 6379
I20260928 02:04:53.440279 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:04:53.440828 6445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:53.441126 1754 external_mini_cluster.cc:1428] Setting key d208bf72b18465b4a0a9ac87cdca80f4
I20260928 02:04:53.449299 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:53.443294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40392 (local address 127.1.182.190:40603)
0928 02:04:53.444220 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:53.444232 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:53.444272 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:04:53.445722 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:53.445742 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:53.446169 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:53.446505 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:53.447643 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:53.447649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:53.447661 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:04:53.447675 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:53.447749 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:04:53.447768 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:53.447934 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:53.448043 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 02:04:53.448315 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":318,"threads_started":1}
I20260928 02:04:53.455121 6445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f: Bootstrap starting.
I20260928 02:04:53.458043 6445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:53.458995 6445 log.cc:824] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:53.461851 6445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f: No bootstrap required, opened a new log
I20260928 02:04:53.468894 6445 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec66c9bb391848df8b98e4f967380d8f" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 40603 } }
I20260928 02:04:53.469246 6445 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:53.469338 6445 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec66c9bb391848df8b98e4f967380d8f, State: Initialized, Role: FOLLOWER
I20260928 02:04:53.469960 6445 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec66c9bb391848df8b98e4f967380d8f" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 40603 } }
I20260928 02:04:53.470185 6445 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:04:53.470312 6445 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:04:53.470501 6445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:53.472580 6445 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec66c9bb391848df8b98e4f967380d8f" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 40603 } }
I20260928 02:04:53.473052 6445 leader_election.cc:302] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec66c9bb391848df8b98e4f967380d8f; no voters:
I20260928 02:04:53.473655 6445 leader_election.cc:288] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:04:53.473809 6450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:53.474916 6450 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f [term 1 LEADER]: Becoming Leader. State: Replica: ec66c9bb391848df8b98e4f967380d8f, State: Running, Role: LEADER
I20260928 02:04:53.475447 6450 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec66c9bb391848df8b98e4f967380d8f" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 40603 } }
I20260928 02:04:53.475884 6445 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:53.478075 6452 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec66c9bb391848df8b98e4f967380d8f. Latest consensus state: current_term: 1 leader_uuid: "ec66c9bb391848df8b98e4f967380d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec66c9bb391848df8b98e4f967380d8f" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 40603 } } }
I20260928 02:04:53.478435 6452 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:53.478179 6451 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec66c9bb391848df8b98e4f967380d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec66c9bb391848df8b98e4f967380d8f" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 40603 } } }
I20260928 02:04:53.478574 6451 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:53.480396 6458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:53.485638 6458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:53.493858 6458 catalog_manager.cc:1347] Generated new cluster ID: 4aabd2f584b349ff860c7336d7e8b371
I20260928 02:04:53.493978 6458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:53.510679 6458 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:04:53.511206 6458 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:53.519515 6458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f: Generated new TSK 0
I20260928 02:04:53.520537 6458 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:04:53.533910 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:40603
--builtin_ntp_servers=127.1.182.148:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:04:53.860183 6469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:53.860469 6469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:53.860561 6469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:53.867851 6469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:53.867956 6469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:53.868072 6469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:04:53.877326 6469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:40603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:53.878573 6469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:53.880537 6469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:53.893857 6477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:53.893906 6475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:53.893817 6474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:53.900892 6469 server_base.cc:1034] running on GCE node
I20260928 02:04:53.901674 6469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:53.902688 6469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:53.903896 6469 hybrid_clock.cc:648] HybridClock initialized: now 1790561093903852 us; error 33 us; skew 500 ppm
I20260928 02:04:53.904263 6469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:53.906994 6469 webserver.cc:460] Webserver started at http://127.1.182.129:40023/ using document root <none> and password file <none>
I20260928 02:04:53.907822 6469 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:53.907938 6469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:53.908344 6469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:53.911500 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "af6a8b7122ca4f3b8872621bb8570b70"
format_stamp: "Formatted at 2026-09-28 02:04:53 on dist-test-slave-23bp"
server_key: "ec83752894223f3974bd146557315d5e"
server_key_iv: "dcdff26bcd9f30bf5b1dd227cee5d29c"
server_key_version: "encryptionkey@0"
I20260928 02:04:53.912313 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "af6a8b7122ca4f3b8872621bb8570b70"
format_stamp: "Formatted at 2026-09-28 02:04:53 on dist-test-slave-23bp"
server_key: "ec83752894223f3974bd146557315d5e"
server_key_iv: "dcdff26bcd9f30bf5b1dd227cee5d29c"
server_key_version: "encryptionkey@0"
I20260928 02:04:53.917763 6469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 02:04:53.921257 6483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:53.922675 6469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:04:53.922857 6469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "af6a8b7122ca4f3b8872621bb8570b70"
format_stamp: "Formatted at 2026-09-28 02:04:53 on dist-test-slave-23bp"
server_key: "ec83752894223f3974bd146557315d5e"
server_key_iv: "dcdff26bcd9f30bf5b1dd227cee5d29c"
server_key_version: "encryptionkey@0"
I20260928 02:04:53.923125 6469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:53.931116 6469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:53.931944 6469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:53.932235 6469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:53.933142 6469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:53.934608 6469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:53.934751 6469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:53.934883 6469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:53.934979 6469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:53.967559 6469 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:35115
I20260928 02:04:53.967615 6595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:35115 every 8 connection(s)
I20260928 02:04:53.968946 6469 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:04:53.973328 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 6469
I20260928 02:04:53.973507 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:04:53.974304 1754 external_mini_cluster.cc:1428] Setting key c6a95f02be0815135e973e4f7d1b7774
I20260928 02:04:53.977468 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:40603
--builtin_ntp_servers=127.1.182.148:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 02:04:53.978384 6598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:53.971564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:56657 (local address 127.1.182.190:40603)
0928 02:04:53.971998 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:53.972003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:53.973387 (+ 1384us) server_negotiation.cc:407] Connection header received
0928 02:04:53.975670 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:53.975674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:53.975783 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:53.975887 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:53.977451 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:53.977459 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:53.977464 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:04:53.977485 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:53.977536 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:04:53.977553 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:53.977679 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:53.977907 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 02:04:53.977971 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":238,"threads_started":1}
I20260928 02:04:53.979324 6599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:53.972146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:40603 (local address 127.1.182.129:56657)
0928 02:04:53.973182 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:53.973240 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:53.975365 (+ 2125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:53.976237 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:53.976250 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:53.976377 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:53.977118 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:04:53.977121 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:53.977157 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:04:53.977283 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:53.977666 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:53.977677 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:53.977688 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 02:04:53.977960 (+ 272us) client_negotiation.cc:240] Negotiation successful
0928 02:04:53.978242 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":262,"threads_started":1}
I20260928 02:04:53.980880 6596 heartbeater.cc:347] Connected to a master server at 127.1.182.190:40603
I20260928 02:04:53.981181 6596 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:53.981751 6596 heartbeater.cc:511] Master 127.1.182.190:40603 requested a full tablet report, sending...
I20260928 02:04:53.984076 6410 ts_manager.cc:194] Registered new tserver with Master: af6a8b7122ca4f3b8872621bb8570b70 (127.1.182.129:35115)
W20260928 02:04:54.296324 6600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:54.296610 6600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:54.296758 6600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:54.304046 6600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:54.304143 6600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:54.304251 6600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:04:54.313313 6600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:40603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:54.314543 6600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:54.316322 6600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:54.329049 6605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:54.329238 6606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:54.329217 6608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:54.329845 6600 server_base.cc:1034] running on GCE node
I20260928 02:04:54.330417 6600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:54.331364 6600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:54.332587 6600 hybrid_clock.cc:648] HybridClock initialized: now 1790561094332558 us; error 52 us; skew 500 ppm
I20260928 02:04:54.332970 6600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:54.335407 6600 webserver.cc:460] Webserver started at http://127.1.182.130:37671/ using document root <none> and password file <none>
I20260928 02:04:54.336107 6600 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:54.336227 6600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:54.336534 6600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:54.339339 6600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "cfee44939488420b984d6e8663e97fcd"
format_stamp: "Formatted at 2026-09-28 02:04:54 on dist-test-slave-23bp"
server_key: "93709a730526ba1d9c6adcb110a4e9d7"
server_key_iv: "5efcce82662b0451c386a0da7d89a872"
server_key_version: "encryptionkey@0"
I20260928 02:04:54.340104 6600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "cfee44939488420b984d6e8663e97fcd"
format_stamp: "Formatted at 2026-09-28 02:04:54 on dist-test-slave-23bp"
server_key: "93709a730526ba1d9c6adcb110a4e9d7"
server_key_iv: "5efcce82662b0451c386a0da7d89a872"
server_key_version: "encryptionkey@0"
I20260928 02:04:54.344936 6600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260928 02:04:54.348219 6614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:54.349493 6600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 02:04:54.349658 6600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "cfee44939488420b984d6e8663e97fcd"
format_stamp: "Formatted at 2026-09-28 02:04:54 on dist-test-slave-23bp"
server_key: "93709a730526ba1d9c6adcb110a4e9d7"
server_key_iv: "5efcce82662b0451c386a0da7d89a872"
server_key_version: "encryptionkey@0"
I20260928 02:04:54.349890 6600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:54.355329 6600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:54.356084 6600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:54.356345 6600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:54.357236 6600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:54.358553 6600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:54.358690 6600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:54.358824 6600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:54.358911 6600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:54.391817 6600 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:38381
I20260928 02:04:54.391907 6726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:38381 every 8 connection(s)
I20260928 02:04:54.393182 6600 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:04:54.397660 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 6600
I20260928 02:04:54.397823 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:04:54.398411 1754 external_mini_cluster.cc:1428] Setting key b95ab0592f0c9037b640f69b3a8ec3fd
I20260928 02:04:54.401454 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:40603
--builtin_ntp_servers=127.1.182.148:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 02:04:54.402246 6598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:54.395776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:46809 (local address 127.1.182.190:40603)
0928 02:04:54.395972 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:54.395977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:54.397812 (+ 1835us) server_negotiation.cc:407] Connection header received
0928 02:04:54.399844 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:54.399850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:54.399902 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:54.400008 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:54.401457 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:54.401461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:54.401464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:04:54.401479 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:54.401507 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:04:54.401518 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:54.401668 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:54.401922 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 02:04:54.401985 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 02:04:54.403228 6729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:54.396315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:40603 (local address 127.1.182.130:46809)
0928 02:04:54.397611 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:54.397671 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:54.399583 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:54.400260 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:54.400279 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:54.400400 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:54.401110 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:04:54.401116 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:54.401168 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:04:54.401285 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:54.401723 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:54.401731 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:54.401737 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:04:54.401999 (+ 262us) client_negotiation.cc:240] Negotiation successful
0928 02:04:54.402273 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":309,"threads_started":1}
I20260928 02:04:54.404569 6727 heartbeater.cc:347] Connected to a master server at 127.1.182.190:40603
I20260928 02:04:54.404852 6727 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:54.405377 6727 heartbeater.cc:511] Master 127.1.182.190:40603 requested a full tablet report, sending...
I20260928 02:04:54.406729 6410 ts_manager.cc:194] Registered new tserver with Master: cfee44939488420b984d6e8663e97fcd (127.1.182.130:38381)
W20260928 02:04:54.727665 6730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:54.727964 6730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:54.728106 6730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:54.735525 6730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:54.735630 6730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:54.735740 6730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:04:54.744983 6730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:40603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:54.746306 6730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:54.748209 6730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:54.760828 6735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:54.760797 6738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:54.760883 6736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:54.761920 6730 server_base.cc:1034] running on GCE node
I20260928 02:04:54.762460 6730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:54.763391 6730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:54.764600 6730 hybrid_clock.cc:648] HybridClock initialized: now 1790561094764544 us; error 46 us; skew 500 ppm
I20260928 02:04:54.764875 6730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:54.767288 6730 webserver.cc:460] Webserver started at http://127.1.182.131:41669/ using document root <none> and password file <none>
I20260928 02:04:54.767941 6730 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:54.768031 6730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:54.768283 6730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:54.771270 6730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "bfe413593746470fb425cf5590ddef22"
format_stamp: "Formatted at 2026-09-28 02:04:54 on dist-test-slave-23bp"
server_key: "6c01d7fec1305619a2602c9e17ecd35b"
server_key_iv: "ecdefe773f681a50ed0e44d065752008"
server_key_version: "encryptionkey@0"
I20260928 02:04:54.772018 6730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "bfe413593746470fb425cf5590ddef22"
format_stamp: "Formatted at 2026-09-28 02:04:54 on dist-test-slave-23bp"
server_key: "6c01d7fec1305619a2602c9e17ecd35b"
server_key_iv: "ecdefe773f681a50ed0e44d065752008"
server_key_version: "encryptionkey@0"
I20260928 02:04:54.777122 6730 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:04:54.780920 6744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:54.782372 6730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:04:54.782511 6730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "bfe413593746470fb425cf5590ddef22"
format_stamp: "Formatted at 2026-09-28 02:04:54 on dist-test-slave-23bp"
server_key: "6c01d7fec1305619a2602c9e17ecd35b"
server_key_iv: "ecdefe773f681a50ed0e44d065752008"
server_key_version: "encryptionkey@0"
I20260928 02:04:54.782729 6730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:54.788245 6730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:54.788964 6730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:54.789180 6730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:54.789985 6730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:04:54.791440 6730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:04:54.791527 6730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:54.791695 6730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:04:54.791786 6730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:54.824573 6730 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:35643
I20260928 02:04:54.824661 6856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:35643 every 8 connection(s)
I20260928 02:04:54.825888 6730 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:04:54.831446 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 6730
I20260928 02:04:54.831626 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:04:54.832062 1754 external_mini_cluster.cc:1428] Setting key 462bfdd4eb1a7c33884a06b43dc6f971
I20260928 02:04:54.834693 6598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:54.828573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:52607 (local address 127.1.182.190:40603)
0928 02:04:54.828753 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:54.828757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:54.830291 (+ 1534us) server_negotiation.cc:407] Connection header received
0928 02:04:54.832168 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:54.832173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:54.832251 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:54.832364 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:54.833950 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:54.833955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:54.833960 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:04:54.833979 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:54.834025 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:04:54.834041 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:54.834172 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:54.834414 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 02:04:54.834460 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:04:54.836058 6859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:54.829099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:40603 (local address 127.1.182.131:52607)
0928 02:04:54.830133 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:54.830176 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:54.831888 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:54.832637 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:54.832651 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:54.832798 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:54.833595 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:04:54.833597 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:54.833637 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:04:54.833722 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:54.834181 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:54.834196 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:54.834207 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 02:04:54.834592 (+ 385us) client_negotiation.cc:240] Negotiation successful
0928 02:04:54.834878 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":282,"threads_started":1}
I20260928 02:04:54.837291 6857 heartbeater.cc:347] Connected to a master server at 127.1.182.190:40603
I20260928 02:04:54.837563 6857 heartbeater.cc:464] Registering TS with master...
I20260928 02:04:54.838078 6857 heartbeater.cc:511] Master 127.1.182.190:40603 requested a full tablet report, sending...
I20260928 02:04:54.839488 6410 ts_manager.cc:194] Registered new tserver with Master: bfe413593746470fb425cf5590ddef22 (127.1.182.131:35643)
I20260928 02:04:54.847858 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:04:54.854672 6598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:54.852174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40406 (local address 127.1.182.190:40603)
0928 02:04:54.852336 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:54.852340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:54.852944 (+ 604us) server_negotiation.cc:407] Connection header received
0928 02:04:54.853090 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:54.853093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:54.853143 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:54.853221 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:54.854196 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:54.854200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:54.854203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:04:54.854216 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:54.854244 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:04:54.854256 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:54.854373 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:54.854423 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 02:04:54.854465 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:04:54.858409 6410 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:04:54.864923 6410 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:04:54.882077 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:54.876624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.1.182.129:35115)
0928 02:04:54.877232 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:54.877241 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:54.877585 (+ 344us) server_negotiation.cc:407] Connection header received
0928 02:04:54.877668 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:54.877677 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:54.877943 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:54.878128 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:54.881084 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:54.881093 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:54.881107 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:04:54.881134 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:54.881195 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:04:54.881218 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:54.881337 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:54.881622 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 02:04:54.881715 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":272,"threads_started":1}
I20260928 02:04:54.882114 6869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:54.876530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:35115 (local address 127.0.0.1:48212)
0928 02:04:54.877259 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:54.877307 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:54.877439 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:54.879355 (+ 1916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:54.879367 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:54.879493 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:54.880778 (+ 1285us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:04:54.880782 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:54.880818 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:04:54.880902 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:54.881389 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:54.881401 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:54.881411 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:04:54.881617 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 02:04:54.881706 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":736,"threads_started":1}
I20260928 02:04:54.886350 6869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:54.878079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:38381 (local address 127.0.0.1:46228)
0928 02:04:54.882508 (+ 4429us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:54.882524 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:54.882613 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:54.883638 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:54.883646 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:54.883670 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:54.884184 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:04:54.884187 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:54.884202 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:04:54.884250 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:54.885906 (+ 1656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:54.885911 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:54.885915 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:04:54.886011 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 02:04:54.886059 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4340,"spinlock_wait_cycles":7808,"thread_start_us":5164,"threads_started":1}
I20260928 02:04:54.886619 6874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:54.878710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46228 (local address 127.1.182.130:38381)
0928 02:04:54.883032 (+ 4322us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:54.883037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:54.883059 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:04:54.883175 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:54.883185 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:54.883416 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:54.883931 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:54.885663 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:54.885668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:54.885676 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:04:54.885691 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:54.885723 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:04:54.885735 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:54.885980 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:54.886058 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 02:04:54.886145 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4188,"thread_start_us":4083,"threads_started":1}
I20260928 02:04:54.886919 6871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:54.877805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:35643 (local address 127.0.0.1:43012)
0928 02:04:54.882498 (+ 4693us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:54.882525 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:54.882613 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:54.884344 (+ 1731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:54.884348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:54.884367 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:54.885040 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:04:54.885043 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:54.885058 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:04:54.885099 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:54.886314 (+ 1215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:54.886318 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:54.886366 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 02:04:54.886485 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 02:04:54.886530 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4542,"spinlock_wait_cycles":9216,"thread_start_us":2647,"threads_started":1}
I20260928 02:04:54.887763 6873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:54.878075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.1.182.131:35643)
0928 02:04:54.883831 (+ 5756us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:54.883864 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:54.883897 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:04:54.883960 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:54.883968 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:54.884172 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:54.884771 (+ 599us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:54.886058 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:54.886066 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:54.886078 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:04:54.886094 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:54.886141 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:04:54.886158 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:54.887201 (+ 1043us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:54.887288 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 02:04:54.887369 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5582,"thread_start_us":5436,"threads_started":1}
I20260928 02:04:54.887442 6531 tablet_service.cc:1467] Processing CreateTablet for tablet 0b244572564a42bd99eaab25e30f2320 (DEFAULT_TABLE table=test-table [id=74dffc67a1ad4000bfd0b02afe9b00fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:54.888831 6531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b244572564a42bd99eaab25e30f2320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:54.889878 6662 tablet_service.cc:1467] Processing CreateTablet for tablet 0b244572564a42bd99eaab25e30f2320 (DEFAULT_TABLE table=test-table [id=74dffc67a1ad4000bfd0b02afe9b00fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:54.890970 6662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b244572564a42bd99eaab25e30f2320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:54.895982 6792 tablet_service.cc:1467] Processing CreateTablet for tablet 0b244572564a42bd99eaab25e30f2320 (DEFAULT_TABLE table=test-table [id=74dffc67a1ad4000bfd0b02afe9b00fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:04:54.897349 6792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b244572564a42bd99eaab25e30f2320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:54.901878 6875 tablet_bootstrap.cc:492] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd: Bootstrap starting.
I20260928 02:04:54.905069 6875 tablet_bootstrap.cc:654] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:54.906038 6875 log.cc:824] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:54.908030 6876 tablet_bootstrap.cc:492] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70: Bootstrap starting.
I20260928 02:04:54.912318 6876 tablet_bootstrap.cc:654] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:54.913350 6875 tablet_bootstrap.cc:492] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd: No bootstrap required, opened a new log
I20260928 02:04:54.913630 6875 ts_tablet_manager.cc:1397] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260928 02:04:54.913904 6876 log.cc:824] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:54.915405 6878 tablet_bootstrap.cc:492] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22: Bootstrap starting.
I20260928 02:04:54.917694 6876 tablet_bootstrap.cc:492] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70: No bootstrap required, opened a new log
I20260928 02:04:54.917961 6876 ts_tablet_manager.cc:1397] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 02:04:54.920499 6878 tablet_bootstrap.cc:654] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:54.921705 6878 log.cc:824] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:54.921454 6875 raft_consensus.cc:348] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af6a8b7122ca4f3b8872621bb8570b70" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35115 } } peers { permanent_uuid: "cfee44939488420b984d6e8663e97fcd" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 38381 } } peers { permanent_uuid: "bfe413593746470fb425cf5590ddef22" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 35643 } }
I20260928 02:04:54.921872 6875 raft_consensus.cc:374] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:54.921967 6875 raft_consensus.cc:729] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfee44939488420b984d6e8663e97fcd, State: Initialized, Role: FOLLOWER
I20260928 02:04:54.922660 6875 consensus_queue.cc:260] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af6a8b7122ca4f3b8872621bb8570b70" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35115 } } peers { permanent_uuid: "cfee44939488420b984d6e8663e97fcd" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 38381 } } peers { permanent_uuid: "bfe413593746470fb425cf5590ddef22" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 35643 } }
I20260928 02:04:54.923806 6727 heartbeater.cc:503] Master 127.1.182.190:40603 was elected leader, sending a full tablet report...
I20260928 02:04:54.924537 6875 ts_tablet_manager.cc:1428] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20260928 02:04:54.925777 6878 tablet_bootstrap.cc:492] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22: No bootstrap required, opened a new log
I20260928 02:04:54.926064 6878 ts_tablet_manager.cc:1397] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260928 02:04:54.928206 6876 raft_consensus.cc:348] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af6a8b7122ca4f3b8872621bb8570b70" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35115 } } peers { permanent_uuid: "cfee44939488420b984d6e8663e97fcd" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 38381 } } peers { permanent_uuid: "bfe413593746470fb425cf5590ddef22" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 35643 } }
I20260928 02:04:54.928673 6876 raft_consensus.cc:374] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:54.928766 6876 raft_consensus.cc:729] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af6a8b7122ca4f3b8872621bb8570b70, State: Initialized, Role: FOLLOWER
I20260928 02:04:54.929381 6876 consensus_queue.cc:260] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af6a8b7122ca4f3b8872621bb8570b70" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35115 } } peers { permanent_uuid: "cfee44939488420b984d6e8663e97fcd" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 38381 } } peers { permanent_uuid: "bfe413593746470fb425cf5590ddef22" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 35643 } }
I20260928 02:04:54.930759 6596 heartbeater.cc:503] Master 127.1.182.190:40603 was elected leader, sending a full tablet report...
I20260928 02:04:54.931238 6876 ts_tablet_manager.cc:1428] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260928 02:04:54.935998 6878 raft_consensus.cc:348] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af6a8b7122ca4f3b8872621bb8570b70" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35115 } } peers { permanent_uuid: "cfee44939488420b984d6e8663e97fcd" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 38381 } } peers { permanent_uuid: "bfe413593746470fb425cf5590ddef22" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 35643 } }
I20260928 02:04:54.936386 6878 raft_consensus.cc:374] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:54.936504 6878 raft_consensus.cc:729] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe413593746470fb425cf5590ddef22, State: Initialized, Role: FOLLOWER
I20260928 02:04:54.937065 6878 consensus_queue.cc:260] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af6a8b7122ca4f3b8872621bb8570b70" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35115 } } peers { permanent_uuid: "cfee44939488420b984d6e8663e97fcd" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 38381 } } peers { permanent_uuid: "bfe413593746470fb425cf5590ddef22" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 35643 } }
I20260928 02:04:54.938084 6857 heartbeater.cc:503] Master 127.1.182.190:40603 was elected leader, sending a full tablet report...
I20260928 02:04:54.938750 6878 ts_tablet_manager.cc:1428] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260928 02:04:54.971170 6597 tablet.cc:2369] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:04:55.077561 6858 tablet.cc:2369] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:04:55.121255 6882 raft_consensus.cc:482] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:04:55.121588 6882 raft_consensus.cc:504] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af6a8b7122ca4f3b8872621bb8570b70" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35115 } } peers { permanent_uuid: "cfee44939488420b984d6e8663e97fcd" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 38381 } } peers { permanent_uuid: "bfe413593746470fb425cf5590ddef22" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 35643 } }
I20260928 02:04:55.123219 6882 leader_election.cc:288] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfee44939488420b984d6e8663e97fcd (127.1.182.130:38381), bfe413593746470fb425cf5590ddef22 (127.1.182.131:35643)
I20260928 02:04:55.126636 6874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:55.123718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:54195 (local address 127.1.182.130:38381)
0928 02:04:55.123931 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:55.123939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:55.124520 (+ 581us) server_negotiation.cc:407] Connection header received
0928 02:04:55.124736 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:55.124746 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:55.124794 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:55.124899 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:55.126021 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:55.126027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:55.126032 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:04:55.126057 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:55.126085 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:04:55.126096 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:55.126225 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:55.126299 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 02:04:55.126358 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:04:55.126982 6873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:55.123728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:33433 (local address 127.1.182.131:35643)
0928 02:04:55.123958 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:55.123968 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:55.124459 (+ 491us) server_negotiation.cc:407] Connection header received
0928 02:04:55.124651 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:55.124656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:55.124723 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:55.124854 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:55.126412 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:55.126418 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:55.126422 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:04:55.126446 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:55.126474 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:04:55.126484 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:55.126620 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:55.126696 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 02:04:55.126756 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:04:55.127336 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:55.123587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:38381 (local address 127.1.182.129:54195)
0928 02:04:55.124436 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:55.124450 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:55.124593 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:55.124930 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:55.124936 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:55.124957 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:55.125833 (+ 876us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:04:55.125835 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:55.125848 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:04:55.125883 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:55.126189 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:55.126194 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:55.126200 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:04:55.126356 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 02:04:55.126402 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":530,"threads_started":1}
I20260928 02:04:55.127463 6884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:55.123519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:35643 (local address 127.1.182.129:33433)
0928 02:04:55.124323 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:55.124342 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:55.124477 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:55.124880 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:55.124885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:55.124914 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:04:55.126207 (+ 1293us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:04:55.126210 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:04:55.126222 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:04:55.126254 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:04:55.126587 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:04:55.126591 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:04:55.126602 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 02:04:55.126743 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 02:04:55.126788 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":461,"threads_started":1}
I20260928 02:04:55.128450 6812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b244572564a42bd99eaab25e30f2320" candidate_uuid: "af6a8b7122ca4f3b8872621bb8570b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe413593746470fb425cf5590ddef22" is_pre_election: true
I20260928 02:04:55.128450 6682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b244572564a42bd99eaab25e30f2320" candidate_uuid: "af6a8b7122ca4f3b8872621bb8570b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfee44939488420b984d6e8663e97fcd" is_pre_election: true
I20260928 02:04:55.129019 6682 raft_consensus.cc:2454] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af6a8b7122ca4f3b8872621bb8570b70 in term 0.
I20260928 02:04:55.129019 6812 raft_consensus.cc:2454] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af6a8b7122ca4f3b8872621bb8570b70 in term 0.
I20260928 02:04:55.129702 6486 leader_election.cc:302] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af6a8b7122ca4f3b8872621bb8570b70, cfee44939488420b984d6e8663e97fcd; no voters:
I20260928 02:04:55.130096 6882 raft_consensus.cc:2781] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:04:55.130242 6882 raft_consensus.cc:482] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:04:55.130343 6882 raft_consensus.cc:3037] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:55.132545 6882 raft_consensus.cc:504] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af6a8b7122ca4f3b8872621bb8570b70" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35115 } } peers { permanent_uuid: "cfee44939488420b984d6e8663e97fcd" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 38381 } } peers { permanent_uuid: "bfe413593746470fb425cf5590ddef22" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 35643 } }
I20260928 02:04:55.133208 6882 leader_election.cc:288] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70 [CANDIDATE]: Term 1 election: Requested vote from peers cfee44939488420b984d6e8663e97fcd (127.1.182.130:38381), bfe413593746470fb425cf5590ddef22 (127.1.182.131:35643)
I20260928 02:04:55.133733 6812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b244572564a42bd99eaab25e30f2320" candidate_uuid: "af6a8b7122ca4f3b8872621bb8570b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe413593746470fb425cf5590ddef22"
I20260928 02:04:55.133719 6682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b244572564a42bd99eaab25e30f2320" candidate_uuid: "af6a8b7122ca4f3b8872621bb8570b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfee44939488420b984d6e8663e97fcd"
I20260928 02:04:55.133970 6682 raft_consensus.cc:3037] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:55.133985 6812 raft_consensus.cc:3037] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:55.137624 6812 raft_consensus.cc:2454] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af6a8b7122ca4f3b8872621bb8570b70 in term 1.
I20260928 02:04:55.137624 6682 raft_consensus.cc:2454] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af6a8b7122ca4f3b8872621bb8570b70 in term 1.
I20260928 02:04:55.138094 6485 leader_election.cc:302] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: af6a8b7122ca4f3b8872621bb8570b70, bfe413593746470fb425cf5590ddef22, cfee44939488420b984d6e8663e97fcd; no voters:
I20260928 02:04:55.138485 6882 raft_consensus.cc:2781] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:55.138929 6882 raft_consensus.cc:686] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70 [term 1 LEADER]: Becoming Leader. State: Replica: af6a8b7122ca4f3b8872621bb8570b70, State: Running, Role: LEADER
I20260928 02:04:55.139487 6882 consensus_queue.cc:237] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af6a8b7122ca4f3b8872621bb8570b70" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35115 } } peers { permanent_uuid: "cfee44939488420b984d6e8663e97fcd" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 38381 } } peers { permanent_uuid: "bfe413593746470fb425cf5590ddef22" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 35643 } }
I20260928 02:04:55.144078 6410 catalog_manager.cc:5495] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70 reported cstate change: term changed from 0 to 1, leader changed from <none> to af6a8b7122ca4f3b8872621bb8570b70 (127.1.182.129). New cstate: current_term: 1 leader_uuid: "af6a8b7122ca4f3b8872621bb8570b70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af6a8b7122ca4f3b8872621bb8570b70" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfee44939488420b984d6e8663e97fcd" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 38381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfe413593746470fb425cf5590ddef22" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 35643 } health_report { overall_health: UNKNOWN } } }
W20260928 02:04:55.145257 6728 tablet.cc:2369] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:04:55.179039 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:55.177095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48218 (local address 127.1.182.129:35115)
0928 02:04:55.177275 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:55.177280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:55.177298 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:04:55.177464 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:55.177467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:55.177550 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:55.177637 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:55.178513 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:55.178518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:55.178521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:04:55.178538 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:55.178573 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:04:55.178586 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:55.178687 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:55.178764 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 02:04:55.178811 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:04:55.187480 6682 raft_consensus.cc:1261] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd [term 1 FOLLOWER]: Refusing update from remote peer af6a8b7122ca4f3b8872621bb8570b70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:04:55.187505 6812 raft_consensus.cc:1261] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22 [term 1 FOLLOWER]: Refusing update from remote peer af6a8b7122ca4f3b8872621bb8570b70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:04:55.188457 6886 consensus_queue.cc:1035] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfe413593746470fb425cf5590ddef22" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 35643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:55.188778 6882 consensus_queue.cc:1035] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfee44939488420b984d6e8663e97fcd" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 38381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:04:55.202936 6891 mvcc.cc:205] Tried to move back new op lower bound from 7334138245877256192 to 7334138245694750720. Current Snapshot: MvccSnapshot[applied={T|T < 7334138245877256192}]
I20260928 02:04:55.213497 6410 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:04:55.213771 6410 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:04:55.217290 6410 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec66c9bb391848df8b98e4f967380d8f: Sending DeleteTablet for 3 replicas of tablet 0b244572564a42bd99eaab25e30f2320
I20260928 02:04:55.218376 6792 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b244572564a42bd99eaab25e30f2320 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:55 UTC) from {username='slave'} at 127.0.0.1:43012
I20260928 02:04:55.218679 6531 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b244572564a42bd99eaab25e30f2320 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:55 UTC) from {username='slave'} at 127.0.0.1:48212
I20260928 02:04:55.219391 6662 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b244572564a42bd99eaab25e30f2320 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:04:55 UTC) from {username='slave'} at 127.0.0.1:46228
I20260928 02:04:55.219745 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 6469
I20260928 02:04:55.233645 6898 tablet_replica.cc:330] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22: stopping tablet replica
I20260928 02:04:55.233994 6900 tablet_replica.cc:330] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd: stopping tablet replica
I20260928 02:04:55.234157 6898 raft_consensus.cc:2229] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:04:55.234390 6900 raft_consensus.cc:2229] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:04:55.234674 6898 raft_consensus.cc:2258] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:55.234829 6900 raft_consensus.cc:2258] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:04:55.236379 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:55.221695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48226 (local address 127.1.182.129:35115)
0928 02:04:55.222254 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:55.222263 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:55.232235 (+ 9972us) server_negotiation.cc:407] Connection header received
0928 02:04:55.232800 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:55.232808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:55.232891 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:55.232978 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:55.235567 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:55.235574 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:55.235580 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:04:55.235605 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:55.235645 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:04:55.235662 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:55.235760 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:55.236026 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 02:04:55.236097 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401}
I20260928 02:04:55.237000 6898 ts_tablet_manager.cc:1905] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:55.237407 6591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:55.237543 6900 ts_tablet_manager.cc:1905] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:04:55.244467 6898 ts_tablet_manager.cc:1918] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:04:55.244630 6898 log.cc:1196] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/0b244572564a42bd99eaab25e30f2320
I20260928 02:04:55.244830 6900 ts_tablet_manager.cc:1918] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:04:55.245012 6900 log.cc:1196] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/0b244572564a42bd99eaab25e30f2320
I20260928 02:04:55.245157 6898 ts_tablet_manager.cc:1939] T 0b244572564a42bd99eaab25e30f2320 P bfe413593746470fb425cf5590ddef22: Deleting consensus metadata
I20260928 02:04:55.245543 6900 ts_tablet_manager.cc:1939] T 0b244572564a42bd99eaab25e30f2320 P cfee44939488420b984d6e8663e97fcd: Deleting consensus metadata
I20260928 02:04:55.247144 6395 catalog_manager.cc:4841] TS bfe413593746470fb425cf5590ddef22 (127.1.182.131:35643): tablet 0b244572564a42bd99eaab25e30f2320 (table test-table [id=74dffc67a1ad4000bfd0b02afe9b00fc]) successfully deleted
I20260928 02:04:55.247136 6396 catalog_manager.cc:4841] TS cfee44939488420b984d6e8663e97fcd (127.1.182.130:38381): tablet 0b244572564a42bd99eaab25e30f2320 (table test-table [id=74dffc67a1ad4000bfd0b02afe9b00fc]) successfully deleted
I20260928 02:04:55.345026 6899 tablet_replica.cc:330] T 0b244572564a42bd99eaab25e30f2320 P af6a8b7122ca4f3b8872621bb8570b70: stopping tablet replica
I20260928 02:04:55.345223 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 6469
W20260928 02:04:55.358760 6396 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.182.129:35115 (error 108)
I20260928 02:04:55.359246 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 6600
I20260928 02:04:55.360371 6871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:55.359855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:35115 (local address 127.0.0.1:48228)
0928 02:04:55.360040 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:55.360182 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:35115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 02:04:55.360671 6396 catalog_manager.cc:4584] TS af6a8b7122ca4f3b8872621bb8570b70 (127.1.182.129:35115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0b244572564a42bd99eaab25e30f2320: Network error: Client connection negotiation failed: client connection to 127.1.182.129:35115: connect: Connection refused (error 111)
I20260928 02:04:55.362062 6874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:55.360077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46232 (local address 127.1.182.130:38381)
0928 02:04:55.360237 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:55.360243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:55.360264 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:04:55.360432 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:55.360437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:55.360546 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:55.360652 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:55.361485 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:55.361489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:55.361492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:04:55.361506 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:55.361532 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:04:55.361543 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:55.361639 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:55.361712 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 02:04:55.361753 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:04:55.362718 6722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:55.414289 6871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:55.413793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:35115 (local address 127.0.0.1:48236)
0928 02:04:55.413980 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:55.414123 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:35115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 02:04:55.415320 6871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:55.414959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:35115 (local address 127.0.0.1:48244)
0928 02:04:55.415125 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:55.415218 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:35115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 02:04:55.452466 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 6600
I20260928 02:04:55.470047 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 6730
I20260928 02:04:55.473157 6873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:55.470851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43014 (local address 127.1.182.131:35643)
0928 02:04:55.471105 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:55.471114 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:55.471137 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:04:55.471358 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:55.471362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:55.471466 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:55.471588 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:55.472437 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:55.472442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:55.472497 (+ 55us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:04:55.472512 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:55.472543 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:04:55.472555 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:55.472694 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:55.472814 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 02:04:55.472872 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 02:04:55.473896 6852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:55.476361 6871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:55.475858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:35115 (local address 127.0.0.1:48250)
0928 02:04:55.476057 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:55.476156 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:35115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 02:04:55.477494 6871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:55.477144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:35115 (local address 127.0.0.1:48260)
0928 02:04:55.477289 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:55.477378 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:35115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 02:04:55.558302 6871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:55.557714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:35115 (local address 127.0.0.1:48262)
0928 02:04:55.557969 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:55.558095 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:35115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260928 02:04:55.559525 6871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:55.559098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:35115 (local address 127.0.0.1:48276)
0928 02:04:55.559289 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:55.559370 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:35115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 02:04:55.562045 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 6730
I20260928 02:04:55.578087 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 6379
I20260928 02:04:55.579319 6440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:55.687438 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 6379
2026-09-28T02:04:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2719 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 02:04:55.729425 1754 mini_oidc.cc:152] Starting JWKS server
I20260928 02:04:55.732368 1754 webserver.cc:460] Webserver started at https://127.0.0.1:37713/ using document root <none> and password file <none>
I20260928 02:04:55.732733 1754 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:04:55.733398 1754 webserver.cc:460] Webserver started at http://127.0.0.1:39179/ using document root <none> and password file <none>
2026-09-28T02:04:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:04:55Z Disabled control of system clock
I20260928 02:04:55.759675 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:45245
--webserver_interface=127.1.182.190
--webserver_port=0
--builtin_ntp_servers=127.1.182.148:41991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:45245
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37713/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561052730740-1754-0/testchainca.cert with env {}
W20260928 02:04:56.081982 6912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:56.082317 6912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:56.082409 6912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:56.089668 6912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:56.089778 6912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:56.089815 6912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:56.089864 6912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:56.089910 6912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:04:56.089969 6912 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:04:56.090009 6912 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37713/jwks.json
I20260928 02:04:56.100050 6912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:41991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:45245
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:45245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37713/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561052730740-1754-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:56.101477 6912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:56.102728 6912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:56.114312 6917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:56.114387 6918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:56.114567 6920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:56.115669 6912 server_base.cc:1034] running on GCE node
I20260928 02:04:56.116423 6912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:56.117798 6912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:56.119004 6912 hybrid_clock.cc:648] HybridClock initialized: now 1790561096118947 us; error 55 us; skew 500 ppm
I20260928 02:04:56.119393 6912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:56.128638 6912 webserver.cc:460] Webserver started at http://127.1.182.190:37801/ using document root <none> and password file <none>
I20260928 02:04:56.129370 6912 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:56.129503 6912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:56.129832 6912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:56.132789 6912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "6486fcbe858c441ea2215cc1d3b3f5cf"
format_stamp: "Formatted at 2026-09-28 02:04:56 on dist-test-slave-23bp"
server_key: "995817a71cfdc43ab20c0accda897cf6"
server_key_iv: "aec2e585798ac7da528ea1b1d36e5138"
server_key_version: "encryptionkey@0"
I20260928 02:04:56.133579 6912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "6486fcbe858c441ea2215cc1d3b3f5cf"
format_stamp: "Formatted at 2026-09-28 02:04:56 on dist-test-slave-23bp"
server_key: "995817a71cfdc43ab20c0accda897cf6"
server_key_iv: "aec2e585798ac7da528ea1b1d36e5138"
server_key_version: "encryptionkey@0"
I20260928 02:04:56.138707 6912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:04:56.142133 6926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:56.143587 6912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:04:56.143771 6912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "6486fcbe858c441ea2215cc1d3b3f5cf"
format_stamp: "Formatted at 2026-09-28 02:04:56 on dist-test-slave-23bp"
server_key: "995817a71cfdc43ab20c0accda897cf6"
server_key_iv: "aec2e585798ac7da528ea1b1d36e5138"
server_key_version: "encryptionkey@0"
I20260928 02:04:56.143989 6912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:56.158594 6912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:56.159459 6912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:56.159775 6912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:56.178104 6912 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:45245
I20260928 02:04:56.178150 6977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:45245 every 8 connection(s)
I20260928 02:04:56.179576 6912 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:04:56.183544 6978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:56.188138 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 6912
I20260928 02:04:56.188356 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:04:56.189141 1754 external_mini_cluster.cc:1428] Setting key b3723d8d36d7ee10982620e6f0a356dc
I20260928 02:04:56.195405 6978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf: Bootstrap starting.
I20260928 02:04:56.198437 6978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:56.199545 6978 log.cc:824] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:56.202646 6978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf: No bootstrap required, opened a new log
I20260928 02:04:56.203230 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:56.191128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.1.182.190:45245)
0928 02:04:56.192098 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:56.192109 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:56.192154 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:04:56.194127 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:56.194152 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:56.194622 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:56.195020 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:56.195650 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:56.196931 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:56.198296 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:56.199059 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:56.199313 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:56.200579 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:56.200595 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:56.200614 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:04:56.201350 (+ 736us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:56.201427 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:04:56.201452 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:56.201543 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:56.201795 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 02:04:56.202161 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":326,"threads_started":1}
I20260928 02:04:56.207191 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:04:56.211020 6978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6486fcbe858c441ea2215cc1d3b3f5cf" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 45245 } }
I20260928 02:04:56.211498 6978 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:56.211609 6978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6486fcbe858c441ea2215cc1d3b3f5cf, State: Initialized, Role: FOLLOWER
I20260928 02:04:56.212199 6978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6486fcbe858c441ea2215cc1d3b3f5cf" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 45245 } }
I20260928 02:04:56.212435 6978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:04:56.212581 6978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:04:56.212745 6978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:56.214987 6978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6486fcbe858c441ea2215cc1d3b3f5cf" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 45245 } }
I20260928 02:04:56.215577 6978 leader_election.cc:302] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6486fcbe858c441ea2215cc1d3b3f5cf; no voters:
I20260928 02:04:56.216162 6978 leader_election.cc:288] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:04:56.216390 6983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:56.217093 6983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf [term 1 LEADER]: Becoming Leader. State: Replica: 6486fcbe858c441ea2215cc1d3b3f5cf, State: Running, Role: LEADER
I20260928 02:04:56.217615 6983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6486fcbe858c441ea2215cc1d3b3f5cf" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 45245 } }
I20260928 02:04:56.218228 6978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:56.220744 6985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6486fcbe858c441ea2215cc1d3b3f5cf. Latest consensus state: current_term: 1 leader_uuid: "6486fcbe858c441ea2215cc1d3b3f5cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6486fcbe858c441ea2215cc1d3b3f5cf" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 45245 } } }
I20260928 02:04:56.221231 6985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:56.221748 6984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6486fcbe858c441ea2215cc1d3b3f5cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6486fcbe858c441ea2215cc1d3b3f5cf" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 45245 } } }
I20260928 02:04:56.222086 6984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:56.222888 6991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:56.228120 6991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:56.237704 6991 catalog_manager.cc:1347] Generated new cluster ID: b2af64ad997c47068942e290a574452d
I20260928 02:04:56.237828 6991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:56.255116 6991 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:04:56.256955 6991 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:56.266108 6991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6486fcbe858c441ea2215cc1d3b3f5cf: Generated new TSK 0
I20260928 02:04:56.267175 6991 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:04:56.318109 7008 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 02:04:56.318301 7008 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:04:56.319410 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:56.290098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56658 (local address 127.1.182.190:45245)
0928 02:04:56.290338 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:56.290345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:56.290775 (+ 430us) server_negotiation.cc:407] Connection header received
0928 02:04:56.291157 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:56.291162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:56.291234 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:56.291359 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:04:56.291990 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:56.292724 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:56.294800 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:56.295296 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:56.295402 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:56.296036 (+ 634us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:04:56.318692 (+ 22656us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:04:56.319056 (+ 364us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56658: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 02:04:56.319554 6981 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56658: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 02:04:56.321240 7007 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:04:56.289988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:45245 (local address 127.0.0.1:56658)
0928 02:04:56.290657 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:56.290680 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:56.290873 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:56.291423 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:56.291429 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:56.291442 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:04:56.291841 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:56.291855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:56.292869 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:56.292875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:56.294654 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:56.294669 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:56.295487 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:56.295501 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:56.295687 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:56.295767 (+ 80us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:04:56.319016 (+ 23249us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:04:56.319425 (+ 409us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.182.190:45245: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":318,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.182.190:45245: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 02:04:56.323022 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 6912
I20260928 02:04:56.324272 6973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:56.415647 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 6912
2026-09-28T02:04:56Z chronyd exiting
I20260928 02:04:56.465612 1754 test_util.cc:175] -----------------------------------------------
I20260928 02:04:56.465750 1754 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561052730740-1754-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (740 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 02:04:56.469355 1754 mini_oidc.cc:152] Starting JWKS server
I20260928 02:04:56.471060 1754 webserver.cc:460] Webserver started at https://127.0.0.1:41347/ using document root <none> and password file <none>
I20260928 02:04:56.471319 1754 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:04:56.471925 1754 webserver.cc:460] Webserver started at http://127.0.0.1:33129/ using document root <none> and password file <none>
2026-09-28T02:04:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:04:56Z Disabled control of system clock
I20260928 02:04:56.498500 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:42047
--webserver_interface=127.1.182.190
--webserver_port=0
--builtin_ntp_servers=127.1.182.148:41489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:42047
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41347/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561052730740-1754-0/testchainca.cert with env {}
W20260928 02:04:56.817145 7015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:56.817431 7015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:56.817517 7015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:56.824656 7015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:56.824770 7015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:56.824811 7015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:56.824867 7015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:56.824911 7015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:04:56.824971 7015 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:04:56.825017 7015 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41347/jwks.json
I20260928 02:04:56.835227 7015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:41489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:42047
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:42047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41347/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561052730740-1754-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:56.836570 7015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:56.837792 7015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:56.849515 7020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:56.849485 7023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:56.849442 7021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:56.850570 7015 server_base.cc:1034] running on GCE node
I20260928 02:04:56.851341 7015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:56.852689 7015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:56.853891 7015 hybrid_clock.cc:648] HybridClock initialized: now 1790561096853857 us; error 28 us; skew 500 ppm
I20260928 02:04:56.854229 7015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:56.856489 7015 webserver.cc:460] Webserver started at http://127.1.182.190:33873/ using document root <none> and password file <none>
I20260928 02:04:56.857156 7015 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:56.857268 7015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:56.857611 7015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:56.860445 7015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "2fff319809fa4c67bac604a108a47f99"
format_stamp: "Formatted at 2026-09-28 02:04:56 on dist-test-slave-23bp"
server_key: "6aad47ce57012794482e90081a78f95e"
server_key_iv: "1a3cb407f25028a6388242fb82c95107"
server_key_version: "encryptionkey@0"
I20260928 02:04:56.861233 7015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "2fff319809fa4c67bac604a108a47f99"
format_stamp: "Formatted at 2026-09-28 02:04:56 on dist-test-slave-23bp"
server_key: "6aad47ce57012794482e90081a78f95e"
server_key_iv: "1a3cb407f25028a6388242fb82c95107"
server_key_version: "encryptionkey@0"
I20260928 02:04:56.866111 7015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:04:56.869380 7029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:56.870694 7015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:04:56.870872 7015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "2fff319809fa4c67bac604a108a47f99"
format_stamp: "Formatted at 2026-09-28 02:04:56 on dist-test-slave-23bp"
server_key: "6aad47ce57012794482e90081a78f95e"
server_key_iv: "1a3cb407f25028a6388242fb82c95107"
server_key_version: "encryptionkey@0"
I20260928 02:04:56.871073 7015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:56.885546 7015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:56.886320 7015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:56.886627 7015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:56.904458 7015 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:42047
I20260928 02:04:56.904551 7080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:42047 every 8 connection(s)
I20260928 02:04:56.905853 7015 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:04:56.907387 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 7015
I20260928 02:04:56.907581 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:04:56.908447 1754 external_mini_cluster.cc:1428] Setting key 40876de47d2b0dbe6204ba223052d374
I20260928 02:04:56.910529 7081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:56.921752 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:56.910283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54182 (local address 127.1.182.190:42047)
0928 02:04:56.911326 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:56.911336 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:56.911378 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:04:56.912904 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:56.912925 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:56.913394 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:56.913710 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:56.914288 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:56.915705 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:56.917512 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:56.918150 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:56.918275 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:56.919532 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:56.919541 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:56.919553 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:04:56.920093 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:56.920168 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:04:56.920191 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:56.920281 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:56.920499 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 02:04:56.920844 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":367,"threads_started":1}
I20260928 02:04:56.923062 7081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99: Bootstrap starting.
I20260928 02:04:56.925673 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:04:56.926347 7081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:56.927354 7081 log.cc:824] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:56.930171 7081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99: No bootstrap required, opened a new log
I20260928 02:04:56.937861 7081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fff319809fa4c67bac604a108a47f99" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 42047 } }
I20260928 02:04:56.938220 7081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:56.938316 7081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fff319809fa4c67bac604a108a47f99, State: Initialized, Role: FOLLOWER
I20260928 02:04:56.938885 7081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fff319809fa4c67bac604a108a47f99" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 42047 } }
I20260928 02:04:56.939131 7081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:04:56.939280 7081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:04:56.939453 7081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:56.941650 7081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fff319809fa4c67bac604a108a47f99" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 42047 } }
I20260928 02:04:56.942186 7081 leader_election.cc:302] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fff319809fa4c67bac604a108a47f99; no voters:
I20260928 02:04:56.942802 7081 leader_election.cc:288] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:04:56.943023 7086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:56.943814 7086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99 [term 1 LEADER]: Becoming Leader. State: Replica: 2fff319809fa4c67bac604a108a47f99, State: Running, Role: LEADER
I20260928 02:04:56.944370 7086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fff319809fa4c67bac604a108a47f99" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 42047 } }
I20260928 02:04:56.944873 7081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:56.947038 7087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fff319809fa4c67bac604a108a47f99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fff319809fa4c67bac604a108a47f99" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 42047 } } }
I20260928 02:04:56.947448 7087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:56.948014 7088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fff319809fa4c67bac604a108a47f99. Latest consensus state: current_term: 1 leader_uuid: "2fff319809fa4c67bac604a108a47f99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fff319809fa4c67bac604a108a47f99" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 42047 } } }
I20260928 02:04:56.948359 7088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:56.948479 7095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:56.955758 7095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:56.965080 7095 catalog_manager.cc:1347] Generated new cluster ID: 955bee37764741c3b0ce7cc7b62c9be5
I20260928 02:04:56.965188 7095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:56.975760 7095 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:04:56.976994 7095 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:56.987447 7095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fff319809fa4c67bac604a108a47f99: Generated new TSK 0
I20260928 02:04:56.988250 7095 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:04:57.020681 7112 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 02:04:57.020823 7112 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:04:57.021829 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:57.001894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54196 (local address 127.1.182.190:42047)
0928 02:04:57.002060 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:57.002065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:57.002567 (+ 502us) server_negotiation.cc:407] Connection header received
0928 02:04:57.002903 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:57.002907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:57.002973 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:57.003185 (+ 212us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:04:57.003742 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:57.004465 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:57.006464 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:57.006901 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:57.006994 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:57.007610 (+ 616us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:04:57.021285 (+ 13675us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41347/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:04:57.021595 (+ 310us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54196: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41347/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 02:04:57.021986 7084 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54196: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41347/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 02:04:57.022272 7111 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:04:57.001711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:42047 (local address 127.0.0.1:54196)
0928 02:04:57.002357 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:57.002383 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:57.002669 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:57.003222 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:57.003230 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:57.003249 (+ 19us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:04:57.003556 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:57.003569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:57.004603 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:57.004609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:57.006320 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:57.006336 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:57.007092 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:57.007100 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:57.007296 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:57.007345 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:04:57.021567 (+ 14222us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41347/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:04:57.021862 (+ 295us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.182.190:42047: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41347/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":345,"threads_started":1}
I20260928 02:04:57.023705 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 7015
I20260928 02:04:57.024899 7076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:57.116452 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 7015
2026-09-28T02:04:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (698 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 02:04:57.168072 1754 mini_oidc.cc:152] Starting JWKS server
I20260928 02:04:57.169843 1754 webserver.cc:460] Webserver started at https://127.0.0.1:38495/ using document root <none> and password file <none>
I20260928 02:04:57.170047 1754 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:04:57.170603 1754 webserver.cc:460] Webserver started at http://127.0.0.1:46087/ using document root <none> and password file <none>
2026-09-28T02:04:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:04:57Z Disabled control of system clock
I20260928 02:04:57.196131 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:44029
--webserver_interface=127.1.182.190
--webserver_port=0
--builtin_ntp_servers=127.1.182.148:40597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:44029
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38495/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:04:57.511175 7119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:57.511520 7119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:57.511687 7119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:57.518778 7119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:57.518932 7119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:57.519034 7119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:57.519164 7119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:57.519250 7119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:04:57.519337 7119 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:04:57.519395 7119 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38495/jwks.json
I20260928 02:04:57.528306 7119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:40597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:44029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:44029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38495/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:57.529621 7119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:57.531416 7119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:57.543160 7124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:57.543403 7127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:57.543404 7125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:57.544569 7119 server_base.cc:1034] running on GCE node
I20260928 02:04:57.545331 7119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:57.546801 7119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:57.548018 7119 hybrid_clock.cc:648] HybridClock initialized: now 1790561097547936 us; error 78 us; skew 500 ppm
I20260928 02:04:57.548384 7119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:57.550627 7119 webserver.cc:460] Webserver started at http://127.1.182.190:34495/ using document root <none> and password file <none>
I20260928 02:04:57.551590 7119 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:57.551728 7119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:57.552104 7119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:57.554929 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "6b63ef0c5ac7442bb9ef42d6d84ed3a8"
format_stamp: "Formatted at 2026-09-28 02:04:57 on dist-test-slave-23bp"
server_key: "e18376fa8458da64993b18c960e93ea1"
server_key_iv: "bf22fdcb2ea2c4afb3b5bd5d80cb6620"
server_key_version: "encryptionkey@0"
I20260928 02:04:57.555801 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "6b63ef0c5ac7442bb9ef42d6d84ed3a8"
format_stamp: "Formatted at 2026-09-28 02:04:57 on dist-test-slave-23bp"
server_key: "e18376fa8458da64993b18c960e93ea1"
server_key_iv: "bf22fdcb2ea2c4afb3b5bd5d80cb6620"
server_key_version: "encryptionkey@0"
I20260928 02:04:57.560791 7119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:04:57.564213 7133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:57.565568 7119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:04:57.565771 7119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "6b63ef0c5ac7442bb9ef42d6d84ed3a8"
format_stamp: "Formatted at 2026-09-28 02:04:57 on dist-test-slave-23bp"
server_key: "e18376fa8458da64993b18c960e93ea1"
server_key_iv: "bf22fdcb2ea2c4afb3b5bd5d80cb6620"
server_key_version: "encryptionkey@0"
I20260928 02:04:57.565995 7119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:57.578657 7119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:57.579485 7119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:57.579787 7119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:57.598132 7119 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:44029
I20260928 02:04:57.598156 7184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:44029 every 8 connection(s)
I20260928 02:04:57.599566 7119 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:04:57.602747 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 7119
I20260928 02:04:57.602981 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:04:57.603303 7185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:57.603689 1754 external_mini_cluster.cc:1428] Setting key cba95cd0ae72f04eb31132e34ac3148b
I20260928 02:04:57.616356 7185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8: Bootstrap starting.
I20260928 02:04:57.617951 7188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:57.605650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.1.182.190:44029)
0928 02:04:57.606845 (+ 1195us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:57.606856 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:57.606895 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:04:57.609091 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:57.609112 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:57.609618 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:57.609932 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:57.610478 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:57.611760 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:57.613676 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:57.614349 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:57.614484 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:57.615764 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:57.615772 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:57.615784 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:04:57.616297 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:57.616372 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:04:57.616395 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:57.616528 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:57.616731 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 02:04:57.617012 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":532,"threads_started":1}
I20260928 02:04:57.619563 7185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:57.620661 7185 log.cc:824] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:57.621766 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:04:57.623413 7185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8: No bootstrap required, opened a new log
I20260928 02:04:57.630790 7185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b63ef0c5ac7442bb9ef42d6d84ed3a8" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44029 } }
I20260928 02:04:57.631170 7185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:57.631269 7185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b63ef0c5ac7442bb9ef42d6d84ed3a8, State: Initialized, Role: FOLLOWER
I20260928 02:04:57.631824 7185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b63ef0c5ac7442bb9ef42d6d84ed3a8" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44029 } }
I20260928 02:04:57.632042 7185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:04:57.632193 7185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:04:57.632368 7185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:57.634585 7185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b63ef0c5ac7442bb9ef42d6d84ed3a8" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44029 } }
I20260928 02:04:57.635181 7185 leader_election.cc:302] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b63ef0c5ac7442bb9ef42d6d84ed3a8; no voters:
I20260928 02:04:57.635895 7185 leader_election.cc:288] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:04:57.636060 7190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:57.636854 7190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8 [term 1 LEADER]: Becoming Leader. State: Replica: 6b63ef0c5ac7442bb9ef42d6d84ed3a8, State: Running, Role: LEADER
I20260928 02:04:57.637432 7190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b63ef0c5ac7442bb9ef42d6d84ed3a8" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44029 } }
I20260928 02:04:57.638015 7185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:57.640362 7191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b63ef0c5ac7442bb9ef42d6d84ed3a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b63ef0c5ac7442bb9ef42d6d84ed3a8" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44029 } } }
I20260928 02:04:57.640800 7191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:57.641408 7192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b63ef0c5ac7442bb9ef42d6d84ed3a8. Latest consensus state: current_term: 1 leader_uuid: "6b63ef0c5ac7442bb9ef42d6d84ed3a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b63ef0c5ac7442bb9ef42d6d84ed3a8" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44029 } } }
I20260928 02:04:57.641758 7192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:57.644232 7197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:57.649202 7197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:57.657766 7197 catalog_manager.cc:1347] Generated new cluster ID: cb5c3bd4582c4ad4b1a3d9a8ae6d3bb8
I20260928 02:04:57.657879 7197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:57.686542 7197 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:04:57.688031 7197 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:57.698386 7197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b63ef0c5ac7442bb9ef42d6d84ed3a8: Generated new TSK 0
I20260928 02:04:57.699499 7197 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:04:57.730319 7188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:57.722128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48874 (local address 127.1.182.190:44029)
0928 02:04:57.722313 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:57.722319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:57.722720 (+ 401us) server_negotiation.cc:407] Connection header received
0928 02:04:57.722915 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:57.722921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:57.723005 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:57.723183 (+ 178us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:04:57.723577 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:57.724692 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:57.726013 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:57.726732 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:57.726847 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:57.727388 (+ 541us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:04:57.729697 (+ 2309us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38495/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:04:57.730014 (+ 317us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48874: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38495/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 02:04:57.730458 7188 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48874: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38495/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 02:04:57.732149 7214 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:04:57.722005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:44029 (local address 127.0.0.1:48874)
0928 02:04:57.722625 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:57.722640 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:57.722754 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:57.723201 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:57.723205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:57.723214 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:04:57.723440 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:57.723447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:57.724813 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:57.724816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:57.725857 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:57.725867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:57.726910 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:57.726914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:57.727031 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:57.727062 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:04:57.729904 (+ 2842us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38495/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:04:57.731732 (+ 1828us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.182.190:44029: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38495/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":1523,"thread_start_us":338,"threads_started":1}
I20260928 02:04:57.733995 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 7119
I20260928 02:04:57.735126 7180 generic_service.cc:196] Checking for leaks (request via RPC)
E20260928 02:04:57.741714 7215 webserver.cc:533] Webserver: sslize failed:
E20260928 02:04:57.741798 7215 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:04:57.825698 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 7119
2026-09-28T02:04:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (712 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T02:04:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:04:57Z Disabled control of system clock
I20260928 02:04:57.906693 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:39543
--webserver_interface=127.1.182.190
--webserver_port=0
--builtin_ntp_servers=127.1.182.148:38565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:39543
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561052730740-1754-0/testchainca.cert with env {}
W20260928 02:04:58.225688 7220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:04:58.225982 7220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:04:58.226081 7220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:04:58.233417 7220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:04:58.233543 7220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:04:58.233583 7220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:04:58.233641 7220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:04:58.233678 7220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:04:58.233739 7220 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:04:58.233779 7220 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 02:04:58.243747 7220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:38565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:39543
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:39543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561052730740-1754-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:04:58.245011 7220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:04:58.246248 7220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:04:58.257591 7228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:58.257603 7225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:04:58.257638 7226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:04:58.258875 7220 server_base.cc:1034] running on GCE node
I20260928 02:04:58.259694 7220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:04:58.261124 7220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:04:58.262349 7220 hybrid_clock.cc:648] HybridClock initialized: now 1790561098262302 us; error 61 us; skew 500 ppm
I20260928 02:04:58.262702 7220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:04:58.265022 7220 webserver.cc:460] Webserver started at http://127.1.182.190:44575/ using document root <none> and password file <none>
I20260928 02:04:58.265718 7220 fs_manager.cc:359] Metadata directory not provided
I20260928 02:04:58.265851 7220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:04:58.266158 7220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:04:58.269011 7220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "c2518d3fcaae4344895c904b26f00141"
format_stamp: "Formatted at 2026-09-28 02:04:58 on dist-test-slave-23bp"
server_key: "9a322a4ee8251606df81abfde6231adf"
server_key_iv: "21ddb5f93f8968633720d873b54f4394"
server_key_version: "encryptionkey@0"
I20260928 02:04:58.269788 7220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "c2518d3fcaae4344895c904b26f00141"
format_stamp: "Formatted at 2026-09-28 02:04:58 on dist-test-slave-23bp"
server_key: "9a322a4ee8251606df81abfde6231adf"
server_key_iv: "21ddb5f93f8968633720d873b54f4394"
server_key_version: "encryptionkey@0"
I20260928 02:04:58.274618 7220 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260928 02:04:58.277964 7234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:04:58.279388 7220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:04:58.279569 7220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "c2518d3fcaae4344895c904b26f00141"
format_stamp: "Formatted at 2026-09-28 02:04:58 on dist-test-slave-23bp"
server_key: "9a322a4ee8251606df81abfde6231adf"
server_key_iv: "21ddb5f93f8968633720d873b54f4394"
server_key_version: "encryptionkey@0"
I20260928 02:04:58.279788 7220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:04:58.305579 7220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:04:58.306579 7220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:04:58.306908 7220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:04:58.325461 7220 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:39543
I20260928 02:04:58.325505 7285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:39543 every 8 connection(s)
I20260928 02:04:58.326876 7220 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:04:58.330753 7286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:04:58.335666 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 7220
I20260928 02:04:58.335844 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:04:58.336638 1754 external_mini_cluster.cc:1428] Setting key b0180064c20f3c2cf5ab81d7cc0930f5
I20260928 02:04:58.342376 7286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141: Bootstrap starting.
I20260928 02:04:58.346292 7286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141: Neither blocks nor log segments found. Creating new log.
I20260928 02:04:58.347411 7286 log.cc:824] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141: Log is configured to *not* fsync() on all Append() calls
I20260928 02:04:58.350230 7286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141: No bootstrap required, opened a new log
I20260928 02:04:58.352101 7289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:58.338585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53964 (local address 127.1.182.190:39543)
0928 02:04:58.339791 (+ 1206us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:58.339806 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:58.339864 (+ 58us) server_negotiation.cc:407] Connection header received
0928 02:04:58.342147 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:58.342175 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:58.342782 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:58.343260 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:04:58.343794 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:58.345799 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:58.347187 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:58.348275 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:58.348498 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:58.349715 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:04:58.349723 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:04:58.349739 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:04:58.350274 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:04:58.350383 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:04:58.350415 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:04:58.350542 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 02:04:58.350783 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 02:04:58.351132 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":444,"threads_started":1}
I20260928 02:04:58.355952 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:04:58.358693 7286 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2518d3fcaae4344895c904b26f00141" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39543 } }
I20260928 02:04:58.359130 7286 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:04:58.359238 7286 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2518d3fcaae4344895c904b26f00141, State: Initialized, Role: FOLLOWER
I20260928 02:04:58.359773 7286 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2518d3fcaae4344895c904b26f00141" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39543 } }
I20260928 02:04:58.359989 7286 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:04:58.360132 7286 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:04:58.360303 7286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:04:58.362632 7286 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2518d3fcaae4344895c904b26f00141" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39543 } }
I20260928 02:04:58.363061 7286 leader_election.cc:302] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2518d3fcaae4344895c904b26f00141; no voters:
I20260928 02:04:58.363704 7286 leader_election.cc:288] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:04:58.363854 7291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:04:58.364527 7291 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141 [term 1 LEADER]: Becoming Leader. State: Replica: c2518d3fcaae4344895c904b26f00141, State: Running, Role: LEADER
I20260928 02:04:58.365096 7291 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2518d3fcaae4344895c904b26f00141" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39543 } }
I20260928 02:04:58.365806 7286 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:04:58.368191 7293 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2518d3fcaae4344895c904b26f00141. Latest consensus state: current_term: 1 leader_uuid: "c2518d3fcaae4344895c904b26f00141" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2518d3fcaae4344895c904b26f00141" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39543 } } }
I20260928 02:04:58.368146 7292 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2518d3fcaae4344895c904b26f00141" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2518d3fcaae4344895c904b26f00141" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39543 } } }
I20260928 02:04:58.368602 7293 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:58.368631 7292 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141 [sys.catalog]: This master's current role is: LEADER
I20260928 02:04:58.370599 7298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:04:58.375942 7298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:04:58.384438 7298 catalog_manager.cc:1347] Generated new cluster ID: 1d82b0a9bad64f859f81a9dab0d652f4
I20260928 02:04:58.384572 7298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:04:58.403667 7298 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:04:58.404861 7298 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:04:58.412472 7298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2518d3fcaae4344895c904b26f00141: Generated new TSK 0
I20260928 02:04:58.413257 7298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:04:58.471491 7289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:04:58.428071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53974 (local address 127.1.182.190:39543)
0928 02:04:58.428263 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:04:58.428272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:04:58.428635 (+ 363us) server_negotiation.cc:407] Connection header received
0928 02:04:58.428901 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:04:58.428905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:04:58.428979 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:04:58.429134 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:04:58.429614 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:58.430336 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:58.431791 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:04:58.432231 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:04:58.432359 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:58.432827 (+ 468us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:04:58.470595 (+ 37768us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:04:58.471107 (+ 512us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53974: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 02:04:58.471642 7315 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:04:58.427962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39543 (local address 127.0.0.1:53974)
0928 02:04:58.428534 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 02:04:58.428549 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:04:58.428707 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:04:58.429149 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:04:58.429155 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:04:58.429166 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:04:58.429456 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:58.429465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:58.430490 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:58.430498 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:58.431641 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:04:58.431652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:04:58.432408 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:04:58.432415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:04:58.432554 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:04:58.432586 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:04:58.470903 (+ 38317us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:04:58.471222 (+ 319us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.182.190:39543: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":302,"threads_started":1}
W20260928 02:04:58.471645 7289 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53974: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260928 02:04:58.473232 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 7220
I20260928 02:04:58.474437 7281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:04:58.570799 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 7220
2026-09-28T02:04:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (724 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (399 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (404 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:04:59 dist-test-slave-23bp krb5kdc[7330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:04:59 dist-test-slave-23bp krb5kdc[7330](info): set up 2 sockets
Sep 28 02:04:59 dist-test-slave-23bp krb5kdc[7330](info): commencing operation
krb5kdc: starting...
W20260928 02:05:00.471623 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:05:00 dist-test-slave-23bp krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:05:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:05:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:05:00.682623 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:37079
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:44809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:37079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:01.001106 7343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:01.001441 7343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:01.001621 7343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:01.008839 7343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:05:01.008986 7343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:01.009095 7343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:01.009198 7343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:05:01.009284 7343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:05:01.019198 7343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:44809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:37079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:37079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:01.020547 7343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:01.021894 7343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:01.033447 7348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:01.033495 7349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:01.033545 7351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:01.034616 7343 server_base.cc:1034] running on GCE node
I20260928 02:05:01.035449 7343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:01.036806 7343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:01.038038 7343 hybrid_clock.cc:648] HybridClock initialized: now 1790561101037998 us; error 48 us; skew 500 ppm
I20260928 02:05:01.038371 7343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:01 dist-test-slave-23bp krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561101, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:01.041376 7343 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:05:01.042850 7343 webserver.cc:460] Webserver started at http://127.1.182.190:41709/ using document root <none> and password file <none>
I20260928 02:05:01.043603 7343 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:01.043752 7343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:01.044077 7343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:01.046976 7343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "35b51f46cbd74552b0f3cb7ead39f645"
format_stamp: "Formatted at 2026-09-28 02:05:01 on dist-test-slave-23bp"
server_key: "61a797bb5f3850932d78b8876c25087b"
server_key_iv: "d9ed12a984db8a1e4d2511689e3cf0d7"
server_key_version: "encryptionkey@0"
I20260928 02:05:01.047798 7343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "35b51f46cbd74552b0f3cb7ead39f645"
format_stamp: "Formatted at 2026-09-28 02:05:01 on dist-test-slave-23bp"
server_key: "61a797bb5f3850932d78b8876c25087b"
server_key_iv: "d9ed12a984db8a1e4d2511689e3cf0d7"
server_key_version: "encryptionkey@0"
I20260928 02:05:01.052879 7343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:05:01.056291 7358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:01.057670 7343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:05:01.057845 7343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "35b51f46cbd74552b0f3cb7ead39f645"
format_stamp: "Formatted at 2026-09-28 02:05:01 on dist-test-slave-23bp"
server_key: "61a797bb5f3850932d78b8876c25087b"
server_key_iv: "d9ed12a984db8a1e4d2511689e3cf0d7"
server_key_version: "encryptionkey@0"
I20260928 02:05:01.058051 7343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:01.071264 7343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:01.075196 7343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:01.075493 7343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:01.094044 7343 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:37079
I20260928 02:05:01.094080 7409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:37079 every 8 connection(s)
I20260928 02:05:01.095558 7343 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:05:01.099303 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:01.102036 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 7343
I20260928 02:05:01.102236 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:05:01.103012 1754 external_mini_cluster.cc:1428] Setting key 4b8dbd9175127ab9075292ad460f2251
I20260928 02:05:01.111868 7410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645: Bootstrap starting.
I20260928 02:05:01.115684 7410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:01.117029 7410 log.cc:824] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:05:01 dist-test-slave-23bp krb5kdc[7330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:05:01.120267 7410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645: No bootstrap required, opened a new log
I20260928 02:05:01.126754 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:01.105098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43764 (local address 127.1.182.190:37079)
0928 02:05:01.105875 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:01.105886 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:01.105925 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:05:01.107782 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:01.107805 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:01.108212 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:01.108542 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:01.110272 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:01.111662 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:01.113418 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:01.114091 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:01.114219 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:01.119423 (+ 5204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:01.119431 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:01.119446 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:01.119486 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:01.122755 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:01.123576 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:01.123582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:01.123588 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:01.123741 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:01.124117 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:01.124120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:01.124122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:01.124631 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:01.124845 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:01.125291 (+ 446us) server_negotiation.cc:299] Negotiation successful
0928 02:05:01.125687 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":380,"threads_started":1}
I20260928 02:05:01.129356 7410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b51f46cbd74552b0f3cb7ead39f645" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37079 } }
I20260928 02:05:01.129840 7410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:01.129935 7410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35b51f46cbd74552b0f3cb7ead39f645, State: Initialized, Role: FOLLOWER
I20260928 02:05:01.130476 7410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b51f46cbd74552b0f3cb7ead39f645" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37079 } }
I20260928 02:05:01.130677 7410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:05:01.130771 7410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:05:01.130887 7410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:01.132985 7410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b51f46cbd74552b0f3cb7ead39f645" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37079 } }
I20260928 02:05:01.133433 7410 leader_election.cc:302] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35b51f46cbd74552b0f3cb7ead39f645; no voters:
I20260928 02:05:01.134021 7410 leader_election.cc:288] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:05:01.134219 7415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:05:01.134946 7415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645 [term 1 LEADER]: Becoming Leader. State: Replica: 35b51f46cbd74552b0f3cb7ead39f645, State: Running, Role: LEADER
I20260928 02:05:01.135562 7415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b51f46cbd74552b0f3cb7ead39f645" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37079 } }
I20260928 02:05:01.136320 7410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:05:01.138666 7416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35b51f46cbd74552b0f3cb7ead39f645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b51f46cbd74552b0f3cb7ead39f645" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37079 } } }
I20260928 02:05:01.139025 7416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:01.138584 7417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35b51f46cbd74552b0f3cb7ead39f645. Latest consensus state: current_term: 1 leader_uuid: "35b51f46cbd74552b0f3cb7ead39f645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b51f46cbd74552b0f3cb7ead39f645" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37079 } } }
I20260928 02:05:01.139889 7417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:01.143610 7424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:05:01.148293 7424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:05:01.156435 7424 catalog_manager.cc:1347] Generated new cluster ID: adbb48c2be66474da2ea67bd2564cab9
I20260928 02:05:01.156541 7424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:05:01.182703 7424 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:05:01.184119 7424 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:05:01.193506 7424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35b51f46cbd74552b0f3cb7ead39f645: Generated new TSK 0
I20260928 02:05:01.194303 7424 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.182.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
I20260928 02:05:01.254739 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:37079
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:44809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:01.581960 7436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:01.582279 7436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:01.582450 7436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:01.589525 7436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:01.589668 7436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:01.589854 7436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:05:01.599321 7436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:44809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:37079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:01.600605 7436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:01.601899 7436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:01.615031 7441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:01.615093 7442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:01.614955 7444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:01.615974 7436 server_base.cc:1034] running on GCE node
I20260928 02:05:01.616570 7436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:01.617463 7436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:01.618702 7436 hybrid_clock.cc:648] HybridClock initialized: now 1790561101618661 us; error 50 us; skew 500 ppm
I20260928 02:05:01.619009 7436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:01 dist-test-slave-23bp krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561101, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:01.627651 7436 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:05:01.629067 7436 webserver.cc:460] Webserver started at http://127.1.182.129:35603/ using document root <none> and password file <none>
I20260928 02:05:01.629743 7436 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:01.629871 7436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:01.630264 7436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:01.633165 7436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "0de6ed31548e4181a53958b1ae5bc5fd"
format_stamp: "Formatted at 2026-09-28 02:05:01 on dist-test-slave-23bp"
server_key: "7c929afc86bbffd56f7b4572d2099c66"
server_key_iv: "15f9b325707d1484ba2e8a6c470dda7d"
server_key_version: "encryptionkey@0"
I20260928 02:05:01.633950 7436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "0de6ed31548e4181a53958b1ae5bc5fd"
format_stamp: "Formatted at 2026-09-28 02:05:01 on dist-test-slave-23bp"
server_key: "7c929afc86bbffd56f7b4572d2099c66"
server_key_iv: "15f9b325707d1484ba2e8a6c470dda7d"
server_key_version: "encryptionkey@0"
I20260928 02:05:01.639289 7436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260928 02:05:01.642892 7451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:01.644263 7436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:05:01.644430 7436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "0de6ed31548e4181a53958b1ae5bc5fd"
format_stamp: "Formatted at 2026-09-28 02:05:01 on dist-test-slave-23bp"
server_key: "7c929afc86bbffd56f7b4572d2099c66"
server_key_iv: "15f9b325707d1484ba2e8a6c470dda7d"
server_key_version: "encryptionkey@0"
I20260928 02:05:01.644639 7436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:01.670653 7436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:01.674568 7436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:01.674865 7436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:01.675858 7436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:01.677169 7436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:01.677244 7436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:01.677399 7436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:01.677464 7436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:01.710275 7436 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:41917
I20260928 02:05:01.710359 7563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:41917 every 8 connection(s)
I20260928 02:05:01.711614 7436 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:05:01.719755 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 7436
I20260928 02:05:01.719951 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:05:01.720788 1754 external_mini_cluster.cc:1428] Setting key 56b8b0d6ac91d5ff45516f58f823b64c
Sep 28 02:05:01 dist-test-slave-23bp krb5kdc[7330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561101, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:05:01.732285 7566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:01.714118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:43069 (local address 127.1.182.190:37079)
0928 02:05:01.714561 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:01.714565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:01.715546 (+ 981us) server_negotiation.cc:407] Connection header received
0928 02:05:01.717283 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:01.717288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:01.717372 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:01.717486 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:01.718992 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:01.719773 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:01.721911 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:01.722368 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:01.722492 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:01.725790 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:01.725798 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:01.725806 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:01.725851 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:01.729083 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:01.730033 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:01.730037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:01.730040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:01.730157 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:01.730617 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:01.730620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:01.730622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:01.730852 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:01.730968 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:01.731858 (+ 890us) server_negotiation.cc:299] Negotiation successful
0928 02:05:01.732031 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":242,"threads_started":1}
I20260928 02:05:01.733763 7567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:01.714567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:37079 (local address 127.1.182.129:43069)
0928 02:05:01.715315 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:01.715363 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:01.717022 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:01.717736 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:01.717744 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:01.718228 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:01.718805 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:01.718821 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:01.719967 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:01.719974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:01.721760 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:01.721777 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:01.722597 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:01.722605 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:01.722827 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:01.723501 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:01.723523 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:01.725568 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:01.729218 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:01.729229 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:01.729254 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:01.729916 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:01.730263 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:01.730271 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:01.730277 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:01.730517 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:01.730974 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:01.730985 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:01.731450 (+ 465us) client_negotiation.cc:769] Sending connection context
0928 02:05:01.731775 (+ 325us) client_negotiation.cc:240] Negotiation successful
0928 02:05:01.732334 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":287,"threads_started":1}
I20260928 02:05:01.735394 7564 heartbeater.cc:347] Connected to a master server at 127.1.182.190:37079
I20260928 02:05:01.735750 7564 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:01.736543 7564 heartbeater.cc:511] Master 127.1.182.190:37079 requested a full tablet report, sending...
I20260928 02:05:01.738807 7375 ts_manager.cc:194] Registered new tserver with Master: 0de6ed31548e4181a53958b1ae5bc5fd (127.1.182.129:41917)
I20260928 02:05:01.740368 7375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:43069
WARNING: no policy specified for kudu/127.1.182.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
I20260928 02:05:01.769800 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:37079
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:44809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:02.093063 7570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:02.093385 7570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:02.093600 7570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:02.100980 7570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:02.101126 7570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:02.101305 7570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:05:02.111238 7570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:44809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:37079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:02.112504 7570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:02.113828 7570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:02.127419 7575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:02.127321 7576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:02.127300 7578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:02.128167 7570 server_base.cc:1034] running on GCE node
I20260928 02:05:02.128726 7570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:02.129655 7570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:02.130854 7570 hybrid_clock.cc:648] HybridClock initialized: now 1790561102130810 us; error 31 us; skew 500 ppm
I20260928 02:05:02.131273 7570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:02 dist-test-slave-23bp krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561102, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:02.134281 7570 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:05:02.135658 7570 webserver.cc:460] Webserver started at http://127.1.182.130:39701/ using document root <none> and password file <none>
I20260928 02:05:02.136304 7570 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:02.136408 7570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:02.136749 7570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:02.139611 7570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "ab773ceba3ed44369c3b940adf70abbf"
format_stamp: "Formatted at 2026-09-28 02:05:02 on dist-test-slave-23bp"
server_key: "124d18a3f47228303363a92049b9a2a2"
server_key_iv: "50b5c0da7d6468055dd5518d2eb239a2"
server_key_version: "encryptionkey@0"
I20260928 02:05:02.140460 7570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "ab773ceba3ed44369c3b940adf70abbf"
format_stamp: "Formatted at 2026-09-28 02:05:02 on dist-test-slave-23bp"
server_key: "124d18a3f47228303363a92049b9a2a2"
server_key_iv: "50b5c0da7d6468055dd5518d2eb239a2"
server_key_version: "encryptionkey@0"
I20260928 02:05:02.145458 7570 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 02:05:02.149102 7585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:02.150413 7570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:05:02.150554 7570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "ab773ceba3ed44369c3b940adf70abbf"
format_stamp: "Formatted at 2026-09-28 02:05:02 on dist-test-slave-23bp"
server_key: "124d18a3f47228303363a92049b9a2a2"
server_key_iv: "50b5c0da7d6468055dd5518d2eb239a2"
server_key_version: "encryptionkey@0"
I20260928 02:05:02.150722 7570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:02.167439 7570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:02.171671 7570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:02.171907 7570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:02.172740 7570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:02.173955 7570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:02.174026 7570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:02.174111 7570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:02.174154 7570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:02.206621 7570 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:40221
I20260928 02:05:02.206658 7697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:40221 every 8 connection(s)
I20260928 02:05:02.207988 7570 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:05:02.208938 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 7570
I20260928 02:05:02.209151 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:05:02.209605 1754 external_mini_cluster.cc:1428] Setting key 38673289de58021a1949830a63938888
Sep 28 02:05:02 dist-test-slave-23bp krb5kdc[7330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561102, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.182.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.131@KRBTEST.COM" created.
I20260928 02:05:02.232404 7566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:02.211821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:55957 (local address 127.1.182.190:37079)
0928 02:05:02.211981 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:02.211988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:02.215326 (+ 3338us) server_negotiation.cc:407] Connection header received
0928 02:05:02.217142 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:02.217148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:02.217239 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:02.217354 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:02.218855 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:02.219609 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:02.221124 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:02.221568 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:02.221701 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:02.225525 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:02.225533 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:02.225539 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:02.225592 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:02.229250 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:02.230154 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:02.230158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:02.230161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:02.230283 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:02.230738 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:02.230741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:02.230743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:02.230999 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:02.231174 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:02.231918 (+ 744us) server_negotiation.cc:299] Negotiation successful
0928 02:05:02.232090 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:05:02.233511 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:02.211336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:37079 (local address 127.1.182.130:55957)
0928 02:05:02.215145 (+ 3809us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:02.215202 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:02.216864 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:02.217606 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:02.217615 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:02.218123 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:02.218687 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:02.218704 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:02.219769 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:02.219773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:02.220978 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:02.220989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:02.221793 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:02.221800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:02.222023 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:02.222707 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:02.222731 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:02.225239 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:02.229399 (+ 4160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:02.229406 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:02.229419 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:02.230047 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:02.230379 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:02.230382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:02.230385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:02.230630 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:02.231161 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:02.231167 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:02.231481 (+ 314us) client_negotiation.cc:769] Sending connection context
0928 02:05:02.231852 (+ 371us) client_negotiation.cc:240] Negotiation successful
0928 02:05:02.232303 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3485,"thread_start_us":3260,"threads_started":1}
I20260928 02:05:02.235031 7698 heartbeater.cc:347] Connected to a master server at 127.1.182.190:37079
I20260928 02:05:02.235394 7698 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:02.236349 7698 heartbeater.cc:511] Master 127.1.182.190:37079 requested a full tablet report, sending...
I20260928 02:05:02.237991 7375 ts_manager.cc:194] Registered new tserver with Master: ab773ceba3ed44369c3b940adf70abbf (127.1.182.130:40221)
I20260928 02:05:02.238897 7375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:55957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
I20260928 02:05:02.258870 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:37079
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:44809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:02.594704 7703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:02.595045 7703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:02.595234 7703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:02.602244 7703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:02.602401 7703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:02.602591 7703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:05:02.612522 7703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:44809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:37079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:02.613795 7703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:02.615159 7703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:02.628095 7708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:02.628147 7709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:02.628163 7711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:02.628952 7703 server_base.cc:1034] running on GCE node
I20260928 02:05:02.629503 7703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:02.630416 7703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:02.631616 7703 hybrid_clock.cc:648] HybridClock initialized: now 1790561102631553 us; error 59 us; skew 500 ppm
I20260928 02:05:02.631934 7703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:02 dist-test-slave-23bp krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561102, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:02.634979 7703 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:05:02.636531 7703 webserver.cc:460] Webserver started at http://127.1.182.131:37377/ using document root <none> and password file <none>
I20260928 02:05:02.637291 7703 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:02.637426 7703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:02.637743 7703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:02.640770 7703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "cdb11bdad0f74cf5b1941031cfe3197c"
format_stamp: "Formatted at 2026-09-28 02:05:02 on dist-test-slave-23bp"
server_key: "b197f99298c38bfbe4364b74e7c92281"
server_key_iv: "0daaa868a26fcc5af9d552a30694aaac"
server_key_version: "encryptionkey@0"
I20260928 02:05:02.641569 7703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "cdb11bdad0f74cf5b1941031cfe3197c"
format_stamp: "Formatted at 2026-09-28 02:05:02 on dist-test-slave-23bp"
server_key: "b197f99298c38bfbe4364b74e7c92281"
server_key_iv: "0daaa868a26fcc5af9d552a30694aaac"
server_key_version: "encryptionkey@0"
I20260928 02:05:02.646852 7703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 02:05:02.650621 7718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:02.651970 7703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:05:02.652135 7703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "cdb11bdad0f74cf5b1941031cfe3197c"
format_stamp: "Formatted at 2026-09-28 02:05:02 on dist-test-slave-23bp"
server_key: "b197f99298c38bfbe4364b74e7c92281"
server_key_iv: "0daaa868a26fcc5af9d552a30694aaac"
server_key_version: "encryptionkey@0"
I20260928 02:05:02.652340 7703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:02.677297 7703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:02.681365 7703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:02.681708 7703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:02.682683 7703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:02.684047 7703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:02.684121 7703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:02.684262 7703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:02.684309 7703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:02.717499 7703 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:35609
I20260928 02:05:02.717530 7830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:35609 every 8 connection(s)
I20260928 02:05:02.718828 7703 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:05:02.728310 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 7703
I20260928 02:05:02.728494 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:05:02.728941 1754 external_mini_cluster.cc:1428] Setting key 9bbdd3b8b2e9a1d1ce1c615ecde308ab
Sep 28 02:05:02 dist-test-slave-23bp krb5kdc[7330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561102, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:05:02.738364 7566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:02.721435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:42827 (local address 127.1.182.190:37079)
0928 02:05:02.721607 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:02.721611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:02.722735 (+ 1124us) server_negotiation.cc:407] Connection header received
0928 02:05:02.724509 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:02.724514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:02.724581 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:02.724671 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:02.726062 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:02.726783 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:02.728608 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:02.729090 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:02.729185 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:02.733101 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:02.733106 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:02.733110 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:02.733140 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:02.735668 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:02.736322 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:02.736326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:02.736328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:02.736485 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:02.736840 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:02.736843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:02.736845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:02.737178 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:02.737307 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:02.737966 (+ 659us) server_negotiation.cc:299] Negotiation successful
0928 02:05:02.738129 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:05:02.739389 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:02.721865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:37079 (local address 127.1.182.131:42827)
0928 02:05:02.722558 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:02.722594 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:02.724253 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:02.724866 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:02.724874 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:02.725345 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:02.725918 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:02.725935 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:02.726906 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:02.726909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:02.728374 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:02.728390 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:02.729311 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:02.729318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:02.729602 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:02.730448 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:02.730476 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:02.732905 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:02.735785 (+ 2880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:02.735792 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:02.735804 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:02.736235 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:02.736578 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:02.736581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:02.736584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:02.736748 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:02.737317 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:02.737322 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:02.737650 (+ 328us) client_negotiation.cc:769] Sending connection context
0928 02:05:02.737867 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 02:05:02.738221 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":267,"threads_started":1}
I20260928 02:05:02.740765 7831 heartbeater.cc:347] Connected to a master server at 127.1.182.190:37079
I20260928 02:05:02.741081 7831 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:02.741782 7831 heartbeater.cc:511] Master 127.1.182.190:37079 requested a full tablet report, sending...
I20260928 02:05:02.743361 7375 ts_manager.cc:194] Registered new tserver with Master: cdb11bdad0f74cf5b1941031cfe3197c (127.1.182.131:35609)
I20260928 02:05:02.743353 7564 heartbeater.cc:503] Master 127.1.182.190:37079 was elected leader, sending a full tablet report...
I20260928 02:05:02.744531 7375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:42827
I20260928 02:05:02.744707 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/test-admin.keytab.
Sep 28 02:05:02 dist-test-slave-23bp krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:02.773214 1754 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 02:05:02.778633 7838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:02.776020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49154 (local address 127.1.182.129:41917)
0928 02:05:02.776709 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:02.776717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:02.776742 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:05:02.776799 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:02.776806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:02.777035 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:02.777207 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:02.778112 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49154: BlockingRecv error: recv got EOF from 127.0.0.1:49154 (error 108)
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":357,"threads_started":1}
W20260928 02:05:02.778643 1754 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:05:02 dist-test-slave-23bp krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:02.780277 1754 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:05:02 dist-test-slave-23bp krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:02.781594 1754 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:05:02 dist-test-slave-23bp krb5kdc[7330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:05:02.796325 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:02.783291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49156 (local address 127.1.182.129:41917)
0928 02:05:02.783463 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:02.783469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:02.783633 (+ 164us) server_negotiation.cc:407] Connection header received
0928 02:05:02.783754 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:02.783757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:02.783811 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:02.783889 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:02.784875 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:02.785797 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:02.786931 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:02.787546 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:02.787644 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:02.791172 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:02.791182 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:02.791197 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:02.791233 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:02.793807 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:02.794478 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:02.794486 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:02.794488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:02.794600 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:02.794903 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:02.794906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:02.794908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:02.795328 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:02.795551 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:02.795849 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 02:05:02.796060 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 02:05:02.801887 7838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:02.800268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49160 (local address 127.1.182.129:41917)
0928 02:05:02.800424 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:02.800428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:02.800780 (+ 352us) server_negotiation.cc:407] Connection header received
0928 02:05:02.800892 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:02.800896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:02.800946 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:02.801028 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:02.801623 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49160: BlockingRecv error: recv got EOF from 127.0.0.1:49160 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 02:05:02.802148 1754 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:05:02 dist-test-slave-23bp krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:02.803764 1754 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:05:02 dist-test-slave-23bp krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:02.805006 1754 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:05:02 dist-test-slave-23bp krb5kdc[7330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:05:02.818783 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:02.806298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49168 (local address 127.1.182.129:41917)
0928 02:05:02.806501 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:02.806506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:02.806737 (+ 231us) server_negotiation.cc:407] Connection header received
0928 02:05:02.806846 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:02.806849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:02.806898 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:02.806989 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:02.807875 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:02.808564 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:02.809993 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:02.810491 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:02.810584 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:02.814143 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:02.814147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:02.814150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:02.814177 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:02.816676 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:02.817214 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:02.817218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:02.817220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:02.817328 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:02.817633 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:02.817637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:02.817638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:02.817884 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:02.818067 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:02.818295 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 02:05:02.818523 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 02:05:02.822918 7838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:02.821314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49182 (local address 127.1.182.129:41917)
0928 02:05:02.821502 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:02.821507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:02.821689 (+ 182us) server_negotiation.cc:407] Connection header received
0928 02:05:02.821738 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:02.821741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:02.821825 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:02.821906 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:02.822697 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49182: BlockingRecv error: recv got EOF from 127.0.0.1:49182 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 02:05:02.823041 1754 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561052730740-1754-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:05:02 dist-test-slave-23bp krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:02.824445 1754 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:05:02 dist-test-slave-23bp krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:02.825666 1754 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:05:02 dist-test-slave-23bp krb5kdc[7330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:05:02.839031 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:02.826784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49188 (local address 127.1.182.129:41917)
0928 02:05:02.826973 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:02.826978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:02.827223 (+ 245us) server_negotiation.cc:407] Connection header received
0928 02:05:02.827329 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:02.827333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:02.827382 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:02.827459 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:02.828320 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:02.829045 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:02.830196 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:02.830755 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:02.830845 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:02.834108 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:02.834113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:02.834116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:02.834143 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:02.836818 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:02.837426 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:02.837429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:02.837431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:02.837573 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:02.837879 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:02.837882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:02.837884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:02.838121 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:02.838330 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:02.838543 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 02:05:02.838723 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:05:02.840597 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 7436
I20260928 02:05:02.852144 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:02.841283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49198 (local address 127.1.182.129:41917)
0928 02:05:02.841432 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:02.841439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:02.841742 (+ 303us) server_negotiation.cc:407] Connection header received
0928 02:05:02.841919 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:02.841930 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:02.841981 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:02.842077 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:02.842950 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:02.843684 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:02.844849 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:02.845283 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:02.845408 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:02.847370 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:02.847375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:02.847378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:02.847405 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:02.849662 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:02.850255 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:02.850262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:02.850266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:02.850397 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:02.850755 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:02.850761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:02.850764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:02.851004 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:02.851284 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:02.851620 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 02:05:02.851829 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:05:02.852681 7559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:02.936507 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 7436
I20260928 02:05:02.952934 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 7570
Sep 28 02:05:02 dist-test-slave-23bp krb5kdc[7330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.130@KRBTEST.COM
I20260928 02:05:02.968020 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:02.953833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.1.182.130:40221)
0928 02:05:02.954349 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:02.954356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:02.954382 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:05:02.954445 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:02.954451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:02.954652 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:02.954779 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:02.955852 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:02.956889 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:02.958110 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:02.958639 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:02.958731 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:02.962705 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:02.962716 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:02.962730 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:02.962762 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:02.965330 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:02.965997 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:02.966004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:02.966007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:02.966125 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:02.966491 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:02.966496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:02.966500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:02.966828 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:02.967166 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:02.967464 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 02:05:02.967714 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":279,"threads_started":1}
I20260928 02:05:02.969244 7693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:03.052528 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 7570
I20260928 02:05:03.068498 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 7703
Sep 28 02:05:03 dist-test-slave-23bp krb5kdc[7330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.131@KRBTEST.COM
I20260928 02:05:03.083957 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:03.069428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.1.182.131:35609)
0928 02:05:03.070127 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:03.070135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:03.070164 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:05:03.070226 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:03.070233 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:03.070443 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:03.070640 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:03.071767 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:03.072779 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:03.074056 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:03.074605 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:03.074740 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:03.078795 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:03.078805 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:03.078819 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:03.078851 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:03.081587 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:03.082239 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:03.082246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:03.082248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:03.082353 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:03.082674 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:03.082677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:03.082679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:03.082991 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:03.083291 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:03.083486 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 02:05:03.083685 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":385,"threads_started":1}
I20260928 02:05:03.085230 7826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:03.168045 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 7703
I20260928 02:05:03.183853 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 7343
I20260928 02:05:03.184978 7405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:03.291435 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 7343
2026-09-28T02:05:03Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3922 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:05:03 dist-test-slave-23bp krb5kdc[7858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:05:03 dist-test-slave-23bp krb5kdc[7858](info): set up 2 sockets
Sep 28 02:05:03 dist-test-slave-23bp krb5kdc[7858](info): commencing operation
krb5kdc: starting...
W20260928 02:05:04.378185 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:05:04 dist-test-slave-23bp krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:05:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:05:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.190.keytab.
Entry for principal oryx/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.190.keytab.
I20260928 02:05:04.590268 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:39651
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.190.keytab
--principal=oryx/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:39651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:04.918633 7871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:04.918983 7871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:04.919178 7871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:04.926589 7871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:05:04.926754 7871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:04.926859 7871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:04.927011 7871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:05:04.927120 7871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:05:04.936995 7871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:39651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.190.keytab
--principal=oryx/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:39651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:04.938436 7871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:04.939834 7871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:04.951481 7877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:04.951514 7876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:04.951447 7879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:04.952740 7871 server_base.cc:1034] running on GCE node
I20260928 02:05:04.953521 7871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:04.954859 7871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:04.956094 7871 hybrid_clock.cc:648] HybridClock initialized: now 1790561104956056 us; error 62 us; skew 500 ppm
I20260928 02:05:04.956439 7871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:04 dist-test-slave-23bp krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561104, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:04.959630 7871 init.cc:377] Logged in from keytab as oryx/127.1.182.190@KRBTEST.COM (short username oryx)
I20260928 02:05:04.961113 7871 webserver.cc:460] Webserver started at http://127.1.182.190:35819/ using document root <none> and password file <none>
I20260928 02:05:04.961786 7871 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:04.961928 7871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:04.962229 7871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:04.965116 7871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "bed972ae228a4bca907f46323a578168"
format_stamp: "Formatted at 2026-09-28 02:05:04 on dist-test-slave-23bp"
server_key: "19e239881d84c2cc7046ffcca5c2594a"
server_key_iv: "b0d0a163e67f90e53750562d8f63ac65"
server_key_version: "encryptionkey@0"
I20260928 02:05:04.965937 7871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "bed972ae228a4bca907f46323a578168"
format_stamp: "Formatted at 2026-09-28 02:05:04 on dist-test-slave-23bp"
server_key: "19e239881d84c2cc7046ffcca5c2594a"
server_key_iv: "b0d0a163e67f90e53750562d8f63ac65"
server_key_version: "encryptionkey@0"
I20260928 02:05:04.970870 7871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 02:05:04.974037 7886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:04.975405 7871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:05:04.975579 7871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "bed972ae228a4bca907f46323a578168"
format_stamp: "Formatted at 2026-09-28 02:05:04 on dist-test-slave-23bp"
server_key: "19e239881d84c2cc7046ffcca5c2594a"
server_key_iv: "b0d0a163e67f90e53750562d8f63ac65"
server_key_version: "encryptionkey@0"
I20260928 02:05:04.975760 7871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:04.991344 7871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:04.994961 7871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:04.995258 7871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:05.013358 7871 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:39651
I20260928 02:05:05.013391 7937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:39651 every 8 connection(s)
I20260928 02:05:05.014748 7871 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:05:05.018476 7938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:05.020948 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 7871
I20260928 02:05:05.021163 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:05:05.021785 1754 external_mini_cluster.cc:1428] Setting key 33c813a237aee8e65a6cd5e68fe87360
I20260928 02:05:05.031092 7938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168: Bootstrap starting.
I20260928 02:05:05.034710 7938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:05.035754 7938 log.cc:824] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:05:05 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM
I20260928 02:05:05.038718 7938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168: No bootstrap required, opened a new log
I20260928 02:05:05.046614 7941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:05.023917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44924 (local address 127.1.182.190:39651)
0928 02:05:05.024635 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:05.024649 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:05.024695 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:05:05.026974 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:05.027002 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:05.027575 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:05.028028 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:05.029337 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:05.031285 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:05.032542 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:05.033609 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:05.033808 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:05.037915 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:05.037926 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:05.037945 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:05.037984 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:05.042103 (+ 4119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:05.042766 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:05.042779 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:05.042792 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:05.042992 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:05.043400 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:05.043406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:05.043409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:05.044084 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:05.044394 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:05.044789 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 02:05:05.045210 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":281,"threads_started":1}
I20260928 02:05:05.048903 7938 raft_consensus.cc:348] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } }
I20260928 02:05:05.049281 7938 raft_consensus.cc:374] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:05.049391 7938 raft_consensus.cc:729] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed972ae228a4bca907f46323a578168, State: Initialized, Role: FOLLOWER
I20260928 02:05:05.049908 7938 consensus_queue.cc:260] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } }
I20260928 02:05:05.050148 7938 raft_consensus.cc:388] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:05:05.050289 7938 raft_consensus.cc:482] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:05:05.050455 7938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:05.052611 7938 raft_consensus.cc:504] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } }
I20260928 02:05:05.053078 7938 leader_election.cc:302] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bed972ae228a4bca907f46323a578168; no voters:
I20260928 02:05:05.053743 7938 leader_election.cc:288] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:05:05.053910 7943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:05:05.054620 7943 raft_consensus.cc:686] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [term 1 LEADER]: Becoming Leader. State: Replica: bed972ae228a4bca907f46323a578168, State: Running, Role: LEADER
I20260928 02:05:05.055220 7943 consensus_queue.cc:237] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } }
I20260928 02:05:05.055896 7938 sys_catalog.cc:564] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:05:05.058421 7944 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bed972ae228a4bca907f46323a578168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } }
I20260928 02:05:05.058993 7944 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:05.058858 7945 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bed972ae228a4bca907f46323a578168. Latest consensus state: current_term: 1 leader_uuid: "bed972ae228a4bca907f46323a578168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } }
I20260928 02:05:05.060317 7945 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:05.062083 7951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:05:05.067688 7951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:05:05.075734 7951 catalog_manager.cc:1347] Generated new cluster ID: 341136e4888648f094f3e854341582e0
I20260928 02:05:05.075834 7951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:05:05.093456 7951 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:05:05.094722 7951 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:05:05.106801 7951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168: Generated new TSK 0
I20260928 02:05:05.107908 7951 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.182.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.182.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.182.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.129.keytab.
Entry for principal oryx/127.1.182.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.129.keytab.
I20260928 02:05:05.159003 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39651
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.129.keytab
--principal=oryx/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:05.493110 7964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:05.493440 7964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:05.493616 7964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:05.500939 7964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:05.501082 7964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:05.501271 7964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:05:05.510855 7964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.129.keytab
--principal=oryx/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:05.512202 7964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:05.513554 7964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:05.526798 7970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:05.526770 7972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:05.526868 7969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:05.527619 7964 server_base.cc:1034] running on GCE node
I20260928 02:05:05.528262 7964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:05.529214 7964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:05.530411 7964 hybrid_clock.cc:648] HybridClock initialized: now 1790561105530357 us; error 52 us; skew 500 ppm
I20260928 02:05:05.530727 7964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:05 dist-test-slave-23bp krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561105, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:05.533980 7964 init.cc:377] Logged in from keytab as oryx/127.1.182.129@KRBTEST.COM (short username oryx)
I20260928 02:05:05.535567 7964 webserver.cc:460] Webserver started at http://127.1.182.129:32911/ using document root <none> and password file <none>
I20260928 02:05:05.536252 7964 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:05.536377 7964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:05.536695 7964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:05.540203 7964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "9d156a26f9e140dcb64782bed8729d55"
format_stamp: "Formatted at 2026-09-28 02:05:05 on dist-test-slave-23bp"
server_key: "0b803846949a275626f90b324b7cd9d5"
server_key_iv: "4ab0a075b676ba1b78928c3dc87d10f7"
server_key_version: "encryptionkey@0"
I20260928 02:05:05.540968 7964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "9d156a26f9e140dcb64782bed8729d55"
format_stamp: "Formatted at 2026-09-28 02:05:05 on dist-test-slave-23bp"
server_key: "0b803846949a275626f90b324b7cd9d5"
server_key_iv: "4ab0a075b676ba1b78928c3dc87d10f7"
server_key_version: "encryptionkey@0"
I20260928 02:05:05.546109 7964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:05:05.549546 7979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:05.550956 7964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:05:05.551182 7964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "9d156a26f9e140dcb64782bed8729d55"
format_stamp: "Formatted at 2026-09-28 02:05:05 on dist-test-slave-23bp"
server_key: "0b803846949a275626f90b324b7cd9d5"
server_key_iv: "4ab0a075b676ba1b78928c3dc87d10f7"
server_key_version: "encryptionkey@0"
I20260928 02:05:05.551388 7964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:05.569727 7964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:05.573987 7964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:05.574268 7964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:05.575209 7964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:05.576504 7964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:05.576584 7964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:05.576741 7964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:05.576800 7964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:05.609845 7964 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:46607
I20260928 02:05:05.609889 8091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:46607 every 8 connection(s)
I20260928 02:05:05.611249 7964 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:05:05.621275 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 7964
I20260928 02:05:05.621480 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:05:05.622226 1754 external_mini_cluster.cc:1428] Setting key 21aa126cbeb00d7c0cd321186156f3ff
Sep 28 02:05:05 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561105, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM
I20260928 02:05:05.631371 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:05.613681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:58259 (local address 127.1.182.190:39651)
0928 02:05:05.614103 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:05.614108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:05.614981 (+ 873us) server_negotiation.cc:407] Connection header received
0928 02:05:05.616831 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:05.616836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:05.616921 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:05.617047 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:05.618482 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:05.619255 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:05.620771 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:05.621403 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:05.621550 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:05.625801 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:05.625807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:05.625812 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:05.625850 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:05.628423 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:05.629112 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:05.629119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:05.629125 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:05.629259 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:05.629684 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:05.629691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:05.629694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:05.629945 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:05.630091 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:05.630763 (+ 672us) server_negotiation.cc:299] Negotiation successful
0928 02:05:05.630969 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":224,"threads_started":1}
I20260928 02:05:05.632349 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:05.614167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.129:58259)
0928 02:05:05.614833 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:05.614869 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:05.616542 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:05.617272 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:05.617281 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:05.617773 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:05.618318 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:05.618334 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:05.619408 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:05.619412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:05.620620 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:05.620632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:05.621706 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:05.621714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:05.621967 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:05.622680 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:05.622706 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:05.625586 (+ 2880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:05.628561 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:05.628570 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:05.628586 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:05.628998 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:05.629380 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:05.629385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:05.629389 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:05.629572 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:05.630067 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:05.630075 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:05.630408 (+ 333us) client_negotiation.cc:769] Sending connection context
0928 02:05:05.630671 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 02:05:05.631225 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":252,"threads_started":1}
I20260928 02:05:05.633922 8092 heartbeater.cc:347] Connected to a master server at 127.1.182.190:39651
I20260928 02:05:05.634245 8092 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:05.635124 8092 heartbeater.cc:511] Master 127.1.182.190:39651 requested a full tablet report, sending...
I20260928 02:05:05.637480 7903 ts_manager.cc:194] Registered new tserver with Master: 9d156a26f9e140dcb64782bed8729d55 (127.1.182.129:46607)
I20260928 02:05:05.639664 7903 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:58259
WARNING: no policy specified for oryx/127.1.182.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.182.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.182.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.130.keytab.
Entry for principal oryx/127.1.182.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.130.keytab.
I20260928 02:05:05.671317 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39651
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.130.keytab
--principal=oryx/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:05.987355 8098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:05.987635 8098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:05.987726 8098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:05.994877 8098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:05.994999 8098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:05.995187 8098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:05:06.005079 8098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.130.keytab
--principal=oryx/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:06.006343 8098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:06.007628 8098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:06.021625 8103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:06.022240 8104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:06.022315 8106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:06.023221 8098 server_base.cc:1034] running on GCE node
I20260928 02:05:06.023857 8098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:06.024932 8098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:06.026175 8098 hybrid_clock.cc:648] HybridClock initialized: now 1790561106026101 us; error 62 us; skew 500 ppm
I20260928 02:05:06.026527 8098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:06 dist-test-slave-23bp krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561106, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:06.030243 8098 init.cc:377] Logged in from keytab as oryx/127.1.182.130@KRBTEST.COM (short username oryx)
I20260928 02:05:06.032042 8098 webserver.cc:460] Webserver started at http://127.1.182.130:41013/ using document root <none> and password file <none>
I20260928 02:05:06.032881 8098 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:06.033041 8098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:06.033362 8098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:06.036684 8098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "5e5d0dbc247b46bc8219f4db2a911ec7"
format_stamp: "Formatted at 2026-09-28 02:05:06 on dist-test-slave-23bp"
server_key: "9f30befbdef3e81c42e675297672ff21"
server_key_iv: "4716a00e2827c6920732d381e725e390"
server_key_version: "encryptionkey@0"
I20260928 02:05:06.037521 8098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "5e5d0dbc247b46bc8219f4db2a911ec7"
format_stamp: "Formatted at 2026-09-28 02:05:06 on dist-test-slave-23bp"
server_key: "9f30befbdef3e81c42e675297672ff21"
server_key_iv: "4716a00e2827c6920732d381e725e390"
server_key_version: "encryptionkey@0"
I20260928 02:05:06.043239 8098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:05:06.046867 8113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:06.048310 8098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:05:06.048534 8098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "5e5d0dbc247b46bc8219f4db2a911ec7"
format_stamp: "Formatted at 2026-09-28 02:05:06 on dist-test-slave-23bp"
server_key: "9f30befbdef3e81c42e675297672ff21"
server_key_iv: "4716a00e2827c6920732d381e725e390"
server_key_version: "encryptionkey@0"
I20260928 02:05:06.048758 8098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:06.095477 8098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:06.100492 8098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:06.100821 8098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:06.101835 8098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:06.103356 8098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:06.103477 8098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:06.103633 8098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:06.103713 8098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:06.139190 8098 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:39733
I20260928 02:05:06.139230 8225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:39733 every 8 connection(s)
I20260928 02:05:06.140538 8098 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:05:06.140976 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 8098
I20260928 02:05:06.141156 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:05:06.141754 1754 external_mini_cluster.cc:1428] Setting key b51a94d1f4d9c23668cc5f035c58d50b
Sep 28 02:05:06 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561106, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.1.182.131@KRBTEST.COM; defaulting to no policy
I20260928 02:05:06.162074 8228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:06.143290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:38857 (local address 127.1.182.190:39651)
0928 02:05:06.143819 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:06.143825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:06.144888 (+ 1063us) server_negotiation.cc:407] Connection header received
0928 02:05:06.146845 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:06.146851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:06.146948 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:06.147074 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:06.148665 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:06.149482 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:06.151218 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:06.151677 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:06.151826 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:06.155711 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:06.155716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:06.155719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:06.155752 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:06.158607 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:06.159526 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:06.159530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:06.159532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:06.159664 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:06.160131 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:06.160142 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:06.160144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:06.160405 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:06.160580 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:06.161388 (+ 808us) server_negotiation.cc:299] Negotiation successful
0928 02:05:06.161664 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":251,"threads_started":1}
I20260928 02:05:06.163357 8229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:06.143946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.130:38857)
0928 02:05:06.144722 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:06.144760 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:06.146561 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:06.147346 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:06.147355 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:06.147883 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:06.148487 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:06.148505 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:06.149650 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:06.149656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:06.150992 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:06.151004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:06.151919 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:06.151929 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:06.152190 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:06.153002 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:06.153024 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:06.155459 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:06.158778 (+ 3319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:06.158787 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:06.158805 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:06.159350 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:06.159783 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:06.159790 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:06.159793 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:06.160000 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:06.160571 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:06.160579 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:06.160923 (+ 344us) client_negotiation.cc:769] Sending connection context
0928 02:05:06.161277 (+ 354us) client_negotiation.cc:240] Negotiation successful
0928 02:05:06.161891 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":313,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.182.131@KRBTEST.COM" created.
I20260928 02:05:06.164901 8226 heartbeater.cc:347] Connected to a master server at 127.1.182.190:39651
I20260928 02:05:06.165228 8226 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:06.165989 8226 heartbeater.cc:511] Master 127.1.182.190:39651 requested a full tablet report, sending...
I20260928 02:05:06.167578 7903 ts_manager.cc:194] Registered new tserver with Master: 5e5d0dbc247b46bc8219f4db2a911ec7 (127.1.182.130:39733)
I20260928 02:05:06.168450 7903 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:38857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.182.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.131.keytab.
Entry for principal oryx/127.1.182.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.131.keytab.
I20260928 02:05:06.191231 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39651
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.131.keytab
--principal=oryx/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:06.511320 8232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:06.511612 8232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:06.511770 8232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:06.518898 8232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:06.519037 8232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:06.519254 8232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:05:06.529042 8232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.131.keytab
--principal=oryx/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:06.530324 8232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:06.531704 8232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:06.544552 8237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:06.544533 8238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:06.544860 8240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:06.545384 8232 server_base.cc:1034] running on GCE node
I20260928 02:05:06.545966 8232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:06.546916 8232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:06.548149 8232 hybrid_clock.cc:648] HybridClock initialized: now 1790561106548103 us; error 57 us; skew 500 ppm
I20260928 02:05:06.548472 8232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:06 dist-test-slave-23bp krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561106, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:06.551643 8232 init.cc:377] Logged in from keytab as oryx/127.1.182.131@KRBTEST.COM (short username oryx)
I20260928 02:05:06.553125 8232 webserver.cc:460] Webserver started at http://127.1.182.131:44767/ using document root <none> and password file <none>
I20260928 02:05:06.553898 8232 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:06.554047 8232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:06.554351 8232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:06.557426 8232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "5db2590c7c594cd9b59e518991ff1445"
format_stamp: "Formatted at 2026-09-28 02:05:06 on dist-test-slave-23bp"
server_key: "0610761f5aab04d393478b83bec709b2"
server_key_iv: "2a6567e45532a2b66ba3ce77cc041a5d"
server_key_version: "encryptionkey@0"
I20260928 02:05:06.558164 8232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "5db2590c7c594cd9b59e518991ff1445"
format_stamp: "Formatted at 2026-09-28 02:05:06 on dist-test-slave-23bp"
server_key: "0610761f5aab04d393478b83bec709b2"
server_key_iv: "2a6567e45532a2b66ba3ce77cc041a5d"
server_key_version: "encryptionkey@0"
I20260928 02:05:06.563309 8232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:05:06.566475 8247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:06.567783 8232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:05:06.567946 8232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "5db2590c7c594cd9b59e518991ff1445"
format_stamp: "Formatted at 2026-09-28 02:05:06 on dist-test-slave-23bp"
server_key: "0610761f5aab04d393478b83bec709b2"
server_key_iv: "2a6567e45532a2b66ba3ce77cc041a5d"
server_key_version: "encryptionkey@0"
I20260928 02:05:06.568157 8232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:06.588946 8232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:06.592798 8232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:06.593079 8232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:06.594003 8232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:06.595283 8232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:06.595398 8232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:06.595554 8232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:06.595605 8232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:06.630736 8232 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:33133
I20260928 02:05:06.630776 8359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:33133 every 8 connection(s)
I20260928 02:05:06.632058 8232 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:05:06.640823 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 8232
I20260928 02:05:06.640997 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:05:06.641465 1754 external_mini_cluster.cc:1428] Setting key 2c3a5c3570812ef9b96da1a994ed2398
I20260928 02:05:06.643440 8092 heartbeater.cc:503] Master 127.1.182.190:39651 was elected leader, sending a full tablet report...
Sep 28 02:05:06 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561106, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM
I20260928 02:05:06.657364 8228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:06.634489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:48697 (local address 127.1.182.190:39651)
0928 02:05:06.634690 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:06.634697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:06.635938 (+ 1241us) server_negotiation.cc:407] Connection header received
0928 02:05:06.637651 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:06.637656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:06.637733 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:06.637822 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:06.639257 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:06.639977 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:06.641574 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:06.642118 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:06.642260 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:06.651277 (+ 9017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:06.651282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:06.651286 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:06.651318 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:06.653916 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:06.654694 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:06.654698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:06.654700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:06.654851 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:06.655405 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:06.655410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:06.655413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:06.655783 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:06.655951 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:06.656782 (+ 831us) server_negotiation.cc:299] Negotiation successful
0928 02:05:06.657028 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:05:06.658449 8362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:06.635029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.131:48697)
0928 02:05:06.635767 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:06.635809 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:06.637401 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:06.638028 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:06.638036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:06.638506 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:06.639045 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:06.639128 (+ 83us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:06.640110 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:06.640113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:06.641427 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:06.641438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:06.642375 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:06.642383 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:06.642669 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:06.648742 (+ 6073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:06.648768 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:06.650994 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:06.654050 (+ 3056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:06.654058 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:06.654073 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:06.654582 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:06.654953 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:06.654956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:06.654958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:06.655237 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:06.655921 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:06.655936 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:06.656355 (+ 419us) client_negotiation.cc:769] Sending connection context
0928 02:05:06.656681 (+ 326us) client_negotiation.cc:240] Negotiation successful
0928 02:05:06.657286 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":279,"threads_started":1}
I20260928 02:05:06.659714 8360 heartbeater.cc:347] Connected to a master server at 127.1.182.190:39651
I20260928 02:05:06.660027 8360 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:06.660708 8360 heartbeater.cc:511] Master 127.1.182.190:39651 requested a full tablet report, sending...
I20260928 02:05:06.662108 7902 ts_manager.cc:194] Registered new tserver with Master: 5db2590c7c594cd9b59e518991ff1445 (127.1.182.131:33133)
I20260928 02:05:06.662961 7902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:48697
I20260928 02:05:06.669031 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.182.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.182.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.182.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.189.keytab.
Entry for principal oryx/127.1.182.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.189.keytab.
I20260928 02:05:06.714718 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.189:39971
--webserver_interface=127.1.182.189
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.189.keytab
--principal=oryx/127.1.182.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:39651,127.1.182.189:39971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:07.034832 8365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:07.035190 8365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:07.035364 8365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:07.042536 8365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:05:07.042690 8365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:07.042802 8365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:07.042909 8365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:05:07.042979 8365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:05:07.052523 8365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:39651,127.1.182.189:39971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.189.keytab
--principal=oryx/127.1.182.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.189:39971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.182.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
W20260928 02:05:07.052919 8365 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.182.190:39651,127.1.182.189:39971'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 02:05:07.053987 8365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:07.055429 8365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:07.067533 8370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:07.067574 8373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:07.067487 8371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:07.068645 8365 server_base.cc:1034] running on GCE node
I20260928 02:05:07.069396 8365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:07.070788 8365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:07.072027 8365 hybrid_clock.cc:648] HybridClock initialized: now 1790561107071952 us; error 63 us; skew 500 ppm
I20260928 02:05:07.072393 8365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:07 dist-test-slave-23bp krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561107, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:07.075459 8365 init.cc:377] Logged in from keytab as oryx/127.1.182.189@KRBTEST.COM (short username oryx)
I20260928 02:05:07.076894 8365 webserver.cc:460] Webserver started at http://127.1.182.189:34617/ using document root <none> and password file <none>
I20260928 02:05:07.077571 8365 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:07.077697 8365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:07.077999 8365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:07.080981 8365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/data/instance:
uuid: "d64d03577e9843debd2679fd7e2b7ec6"
format_stamp: "Formatted at 2026-09-28 02:05:07 on dist-test-slave-23bp"
server_key: "b974a5e31ed150fa7245250f03d844e9"
server_key_iv: "ea1fc5dfd9fa7476607d94b46dec5b94"
server_key_version: "encryptionkey@0"
I20260928 02:05:07.081744 8365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/wal/instance:
uuid: "d64d03577e9843debd2679fd7e2b7ec6"
format_stamp: "Formatted at 2026-09-28 02:05:07 on dist-test-slave-23bp"
server_key: "b974a5e31ed150fa7245250f03d844e9"
server_key_iv: "ea1fc5dfd9fa7476607d94b46dec5b94"
server_key_version: "encryptionkey@0"
I20260928 02:05:07.086620 8365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:05:07.089926 8380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:07.091280 8365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 02:05:07.091463 8365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/wal
uuid: "d64d03577e9843debd2679fd7e2b7ec6"
format_stamp: "Formatted at 2026-09-28 02:05:07 on dist-test-slave-23bp"
server_key: "b974a5e31ed150fa7245250f03d844e9"
server_key_iv: "ea1fc5dfd9fa7476607d94b46dec5b94"
server_key_version: "encryptionkey@0"
I20260928 02:05:07.091677 8365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:07.121637 8365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:07.125842 8365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:07.126104 8365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:07.144716 8365 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.189:39971
I20260928 02:05:07.144739 8431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.189:39971 every 8 connection(s)
I20260928 02:05:07.145998 8365 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/data/info.pb
I20260928 02:05:07.149613 8432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:07.154695 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 8365
I20260928 02:05:07.157181 8432 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:05:07 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561107, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.189@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM
Sep 28 02:05:07 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.182.189@KRBTEST.COM
I20260928 02:05:07.171208 8226 heartbeater.cc:503] Master 127.1.182.190:39651 was elected leader, sending a full tablet report...
I20260928 02:05:07.177591 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.159910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44936 (local address 127.1.182.190:39651)
0928 02:05:07.161178 (+ 1268us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:07.161188 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:07.161206 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:05:07.161267 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:07.161272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:07.161357 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:07.161493 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:07.163310 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.164101 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.166174 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.166562 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.166692 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.170257 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:07.170261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:07.170264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:07.170295 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:07.174028 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.174836 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.174840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.174842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.174962 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.175495 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.175498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.175500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.175868 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:07.176034 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:07.176766 (+ 732us) server_negotiation.cc:299] Negotiation successful
0928 02:05:07.177045 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1144,"thread_start_us":1042,"threads_started":1}
I20260928 02:05:07.178412 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.156920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41854 (local address 127.1.182.189:39971)
0928 02:05:07.157880 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:07.157893 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:07.157946 (+ 53us) server_negotiation.cc:407] Connection header received
0928 02:05:07.159911 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:07.159947 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:07.160609 (+ 662us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:07.160986 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:07.162939 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.164178 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.165878 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.166396 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.166580 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.170726 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:07.170736 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:07.170752 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:07.170791 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:07.174319 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.175098 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.175107 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.175110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.175264 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.175762 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.175766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.175768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.176331 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:07.176567 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:07.176977 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 02:05:07.177352 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":692,"thread_start_us":427,"threads_started":1}
I20260928 02:05:07.178454 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.159869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.0.0.1:44936)
0928 02:05:07.160555 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:07.160597 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:07.160758 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:07.161742 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:07.161748 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:07.162566 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:07.163135 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.163156 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.164244 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.164251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.166018 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.166029 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.166790 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.166798 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.166991 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.167627 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:07.167648 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:07.170068 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:07.174152 (+ 4084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.174161 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.174171 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.174711 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.175112 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.175116 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.175120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.175385 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.176004 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:07.176011 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:07.176358 (+ 347us) client_negotiation.cc:769] Sending connection context
0928 02:05:07.176679 (+ 321us) client_negotiation.cc:240] Negotiation successful
0928 02:05:07.177211 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":41,"spinlock_wait_cycles":41856,"thread_start_us":307,"threads_started":1}
I20260928 02:05:07.180930 8432 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:05:07 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561107, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.189@KRBTEST.COM for oryx/127.1.182.189@KRBTEST.COM
I20260928 02:05:07.193065 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.181744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39971 (local address 127.0.0.1:41860)
0928 02:05:07.181914 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:07.181941 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:07.182041 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:07.182322 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:07.182326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:07.182663 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:07.182784 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.182791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.183810 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.183815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.184657 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.184667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.185319 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.185323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.185537 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.186013 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:07.186037 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:07.187976 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:07.190665 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.190673 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.190677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.191050 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.191436 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.191440 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.191445 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.191587 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.192059 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:07.192064 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:07.192205 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 02:05:07.192337 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 02:05:07.192629 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":3}
I20260928 02:05:07.193069 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.181802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41860 (local address 127.1.182.189:39971)
0928 02:05:07.181927 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:07.181933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:07.181987 (+ 54us) server_negotiation.cc:407] Connection header received
0928 02:05:07.182151 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:07.182156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:07.182215 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:07.182308 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:07.182906 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.183675 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.184792 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.185182 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.185311 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.188143 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:07.188149 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:07.188153 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:07.188189 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:07.190535 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.191192 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.191199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.191202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.191329 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.191687 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.191694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.191697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.191951 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:07.192095 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:07.192423 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 02:05:07.192754 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":160,"server-negotiator.queue_time_us":51}
I20260928 02:05:07.197945 8432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6: Bootstrap starting.
I20260928 02:05:07.201066 8432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:07.202095 8432 log.cc:824] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:05:07.204639 8432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6: No bootstrap required, opened a new log
I20260928 02:05:07.211560 8432 raft_consensus.cc:348] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } }
I20260928 02:05:07.211880 8432 raft_consensus.cc:374] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:07.211974 8432 raft_consensus.cc:729] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d64d03577e9843debd2679fd7e2b7ec6, State: Initialized, Role: FOLLOWER
I20260928 02:05:07.212553 8432 consensus_queue.cc:260] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } }
I20260928 02:05:07.213370 8439 sys_catalog.cc:455] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } } }
I20260928 02:05:07.213639 8439 sys_catalog.cc:458] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:05:07.214365 8432 sys_catalog.cc:564] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:05:07.222589 8450 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:05:07.222721 8450 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:05:07.225167 8365 master_runner.cc:428] Detected that this master d64d03577e9843debd2679fd7e2b7ec6 is joining an existing cluster
I20260928 02:05:07.225261 8365 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.182.189:39971
I20260928 02:05:07.245051 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.228943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.1.182.190:39651)
0928 02:05:07.229120 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:07.229127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:07.229799 (+ 672us) server_negotiation.cc:407] Connection header received
0928 02:05:07.230012 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:07.230017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:07.230117 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:07.230219 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:07.231337 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.232473 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.234540 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.235211 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.235308 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.237806 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:07.237812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:07.237816 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:07.237854 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:07.242029 (+ 4175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.243019 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.243027 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.243031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.243191 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.243588 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.243593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.243597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.243891 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:07.244043 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:07.244462 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 02:05:07.244701 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:05:07.245167 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.228949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.0.0.1:44946)
0928 02:05:07.229688 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:07.229715 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:07.229868 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:07.230238 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:07.230242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:07.230747 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:07.231122 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.231137 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.232655 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.232660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.234368 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.234377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.235603 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.235609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.235786 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.236453 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:07.236476 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:07.237619 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:07.242176 (+ 4557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.242181 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.242185 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.242867 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.243308 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.243313 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.243316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.243459 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.244030 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:07.244034 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:07.244212 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 02:05:07.244366 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 02:05:07.244703 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":432,"threads_started":1}
I20260928 02:05:07.247089 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.228952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41864 (local address 127.1.182.189:39971)
0928 02:05:07.229120 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:07.229129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:07.231658 (+ 2529us) server_negotiation.cc:407] Connection header received
0928 02:05:07.231853 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:07.231858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:07.231913 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:07.231967 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:07.233340 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.234377 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.236649 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.237397 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.237511 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.240447 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:07.240454 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:07.240458 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:07.240504 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:07.244343 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.245102 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.245107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.245113 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.245230 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.245612 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.245617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.245622 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.245893 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:07.246065 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:07.246384 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 02:05:07.246729 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":145,"server-negotiator.queue_time_us":51}
I20260928 02:05:07.247126 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.229513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39971 (local address 127.0.0.1:41864)
0928 02:05:07.231579 (+ 2066us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:07.231596 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:07.231738 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:07.232126 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:07.232131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:07.232841 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:07.233197 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.233208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.234608 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.234613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.236486 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.236498 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.237789 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.237795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.237975 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.238844 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:07.238871 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:07.240269 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:07.244485 (+ 4216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.244490 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.244494 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.244999 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.245345 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.245348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.245351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.245492 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.246027 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:07.246031 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:07.246163 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 02:05:07.246296 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 02:05:07.246590 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1943,"mutex_wait_us":50,"thread_start_us":1686,"threads_started":1}
WARNING: no policy specified for oryx/127.1.182.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.182.188@KRBTEST.COM" created.
Sep 28 02:05:07 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561104, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.190@KRBTEST.COM for oryx/127.1.182.189@KRBTEST.COM
I20260928 02:05:07.274466 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.252342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41868 (local address 127.1.182.189:39971)
0928 02:05:07.252553 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:07.252559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:07.255858 (+ 3299us) server_negotiation.cc:407] Connection header received
0928 02:05:07.256241 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:07.256249 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:07.256372 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:07.256502 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:07.257887 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.258658 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.260614 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.261531 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.261686 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.266311 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:07.266320 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:07.266328 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:07.266369 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:07.270687 (+ 4318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.271660 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.271669 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.271675 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.271911 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.272392 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.272400 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.272406 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.272928 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:07.273075 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:07.273731 (+ 656us) server_negotiation.cc:299] Negotiation successful
0928 02:05:07.274001 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:05:07.275646 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.252258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39971 (local address 127.0.0.1:41868)
0928 02:05:07.255718 (+ 3460us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:07.255763 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:07.256021 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:07.256558 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:07.256567 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:07.257430 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:07.257723 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.257736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.258784 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.258790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.260442 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.260457 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.261952 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.261961 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.262198 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.263032 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:07.263060 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:07.266098 (+ 3038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:07.270842 (+ 4744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.270857 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.270868 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.271528 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.272073 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.272080 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.272087 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.272263 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.273184 (+ 921us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:07.273195 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:07.273396 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 02:05:07.274670 (+ 1274us) client_negotiation.cc:240] Negotiation successful
0928 02:05:07.275051 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3202,"thread_start_us":3013,"threads_started":1}
I20260928 02:05:07.278108 7902 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.182.189:39971: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6"
member_type: NON_VOTER
last_known_addr {
host: "127.1.182.189"
port: 39971
}
attrs {
promote: true
}
}
dest_uuid: "bed972ae228a4bca907f46323a578168"
cas_config_opid_index: -1
I20260928 02:05:07.279356 7902 consensus_queue.cc:237] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } }
I20260928 02:05:07.286479 8462 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d64d03577e9843debd2679fd7e2b7ec6 (127.1.182.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } } }
I20260928 02:05:07.288117 8407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:07.290508 8462 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.182.189:39971
I20260928 02:05:07.290992 8462 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bed972ae228a4bca907f46323a578168" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } } }
I20260928 02:05:07.291253 8462 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:07.292248 8365 master.cc:537] Master@127.1.182.189:39971 shutting down...
I20260928 02:05:07.292975 8407 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 1 FOLLOWER]: Refusing update from remote peer bed972ae228a4bca907f46323a578168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 02:05:07.294103 8462 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.182.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.188.keytab.
Entry for principal oryx/127.1.182.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.188.keytab.
I20260928 02:05:07.297024 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.188:45051
--webserver_interface=127.1.182.188
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.188.keytab
--principal=oryx/127.1.182.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:39651,127.1.182.189:39971,127.1.182.188:45051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:07.299779 7888 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 02:05:07.306465 7888 consensus_peers.cc:487] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 -> Peer d64d03577e9843debd2679fd7e2b7ec6 (127.1.182.189:39971): Couldn't send request to peer d64d03577e9843debd2679fd7e2b7ec6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 02:05:07.308007 8365 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:05:07.308493 8365 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:05:07.308689 8365 tablet_replica.cc:330] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6: stopping tablet replica
I20260928 02:05:07.315172 8365 master.cc:552] Master@127.1.182.189:39971 shutdown complete.
I20260928 02:05:07.315284 8365 master_runner.cc:305] Clearing existing system tablet
I20260928 02:05:07.315784 8365 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:07.315897 8365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:07.318812 8365 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:05:07.320472 8468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:07.320835 8365 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:07.321148 8365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:05:07.321249 8365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/wal
uuid: "d64d03577e9843debd2679fd7e2b7ec6"
format_stamp: "Formatted at 2026-09-28 02:05:07 on dist-test-slave-23bp"
server_key: "b974a5e31ed150fa7245250f03d844e9"
server_key_iv: "ea1fc5dfd9fa7476607d94b46dec5b94"
server_key_version: "encryptionkey@0"
I20260928 02:05:07.324808 8365 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:07.330238 8365 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:05:07.330432 8365 log.cc:1196] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 02:05:07.330844 8365 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6: Deleting consensus metadata
I20260928 02:05:07.331180 8365 master_runner.cc:315] Copying system tablet from 127.1.182.190:39651
I20260928 02:05:07.336761 8365 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6: tablet copy: Beginning tablet copy session from remote peer at address 127.1.182.190:39651
I20260928 02:05:07.350157 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.337517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44952 (local address 127.1.182.190:39651)
0928 02:05:07.337708 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:07.337715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:07.338277 (+ 562us) server_negotiation.cc:407] Connection header received
0928 02:05:07.338439 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:07.338445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:07.338537 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:07.338655 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:07.339586 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.340568 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.341926 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.342391 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.342527 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.344487 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:07.344493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:07.344503 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:07.344537 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:07.347395 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.348076 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.348087 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.348090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.348222 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.348646 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.348651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.348661 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.348943 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:07.349108 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:07.349516 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 02:05:07.349738 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:05:07.350246 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.337498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.0.0.1:44952)
0928 02:05:07.338067 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:07.338083 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:07.338259 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:07.338680 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:07.338685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:07.339194 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:07.339431 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.339441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.340717 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.340722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.341749 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.341762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.342553 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.342559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.342754 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.343478 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:07.343497 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:07.344258 (+ 761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:07.347541 (+ 3283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.347547 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.347551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.347941 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.348337 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.348342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.348347 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.348524 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.349073 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:07.349079 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:07.349261 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 02:05:07.349424 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 02:05:07.349738 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":317,"threads_started":1}
I20260928 02:05:07.351368 7923 tablet_copy_service.cc:140] P bed972ae228a4bca907f46323a578168: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d64d03577e9843debd2679fd7e2b7ec6 ({username='oryx', principal='oryx/127.1.182.189@KRBTEST.COM'} at 127.0.0.1:44952)
I20260928 02:05:07.351652 7923 tablet_copy_service.cc:161] P bed972ae228a4bca907f46323a578168: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d64d03577e9843debd2679fd7e2b7ec6 at {username='oryx', principal='oryx/127.1.182.189@KRBTEST.COM'} at 127.0.0.1:44952: session id = d64d03577e9843debd2679fd7e2b7ec6-00000000000000000000000000000000
I20260928 02:05:07.355237 7923 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:05:07.357482 8365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:05:07.362898 8365 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6: tablet copy: Starting download of 0 data blocks...
I20260928 02:05:07.363248 8365 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6: tablet copy: Starting download of 1 WAL segments...
I20260928 02:05:07.366168 8365 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:05:07.369779 7923 tablet_copy_service.cc:342] P bed972ae228a4bca907f46323a578168: Request end of tablet copy session d64d03577e9843debd2679fd7e2b7ec6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.182.189@KRBTEST.COM'} at 127.0.0.1:44952
I20260928 02:05:07.369987 7923 tablet_copy_service.cc:433] P bed972ae228a4bca907f46323a578168: ending tablet copy session d64d03577e9843debd2679fd7e2b7ec6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d64d03577e9843debd2679fd7e2b7ec6
I20260928 02:05:07.380568 8365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:07.383431 8475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:07.388856 8476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:07.396080 8478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:07.397089 8365 server_base.cc:1034] running on GCE node
I20260928 02:05:07.397423 8365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:07.398077 8365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:07.399284 8365 hybrid_clock.cc:648] HybridClock initialized: now 1790561107399200 us; error 66 us; skew 500 ppm
I20260928 02:05:07.399518 8365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:07 dist-test-slave-23bp krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561107, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:07.401384 8365 init.cc:377] Logged in from keytab as oryx/127.1.182.189@KRBTEST.COM (short username oryx)
I20260928 02:05:07.407997 8365 webserver.cc:460] Webserver started at http://127.1.182.189:37045/ using document root <none> and password file <none>
I20260928 02:05:07.408331 8365 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:07.408425 8365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:07.410945 8365 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:05:07.412613 8485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:07.413049 8365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:05:07.413203 8365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/wal
uuid: "d64d03577e9843debd2679fd7e2b7ec6"
format_stamp: "Formatted at 2026-09-28 02:05:07 on dist-test-slave-23bp"
server_key: "b974a5e31ed150fa7245250f03d844e9"
server_key_iv: "ea1fc5dfd9fa7476607d94b46dec5b94"
server_key_version: "encryptionkey@0"
I20260928 02:05:07.413354 8365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:07.445879 8365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:07.449335 8365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:07.475808 8365 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.189:39971
I20260928 02:05:07.475832 8536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.189:39971 every 8 connection(s)
I20260928 02:05:07.476740 8365 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-1/data/info.pb
I20260928 02:05:07.483644 8537 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:05:07.486171 8537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6: Bootstrap starting.
I20260928 02:05:07.500020 8537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:05:07.500491 8537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6: Bootstrap complete.
I20260928 02:05:07.501485 8537 raft_consensus.cc:348] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } }
I20260928 02:05:07.501706 8537 raft_consensus.cc:729] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d64d03577e9843debd2679fd7e2b7ec6, State: Initialized, Role: LEARNER
I20260928 02:05:07.501924 8537 consensus_queue.cc:260] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } }
I20260928 02:05:07.502828 8540 sys_catalog.cc:455] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } } }
I20260928 02:05:07.503127 8537 sys_catalog.cc:564] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:05:07.503273 8540 sys_catalog.cc:458] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:05:07.515146 8551 catalog_manager.cc:1259] Loaded cluster ID: 341136e4888648f094f3e854341582e0
I20260928 02:05:07.515251 8551 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6: loading cluster ID for follower catalog manager: success
I20260928 02:05:07.517709 8551 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6: acquiring CA information for follower catalog manager: success
I20260928 02:05:07.519541 8551 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:05:07.665539 8360 heartbeater.cc:503] Master 127.1.182.190:39651 was elected leader, sending a full tablet report...
W20260928 02:05:07.736992 8466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:07.737277 8466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:07.737428 8466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:07.744755 8466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:05:07.744859 8466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:07.744951 8466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:07.745046 8466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:05:07.745131 8466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:05:07.756095 8466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:39651,127.1.182.189:39971,127.1.182.188:45051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.188.keytab
--principal=oryx/127.1.182.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.188:45051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.182.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:07.757519 8466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:07.758853 8466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:07.771452 8556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:07.771571 8557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:07.771786 8559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:07.772571 8466 server_base.cc:1034] running on GCE node
I20260928 02:05:07.773473 8466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:07.774964 8466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:07.776197 8466 hybrid_clock.cc:648] HybridClock initialized: now 1790561107776112 us; error 49 us; skew 500 ppm
I20260928 02:05:07.776577 8466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:07 dist-test-slave-23bp krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561107, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:07.779729 8466 init.cc:377] Logged in from keytab as oryx/127.1.182.188@KRBTEST.COM (short username oryx)
I20260928 02:05:07.781436 8466 webserver.cc:460] Webserver started at http://127.1.182.188:41861/ using document root <none> and password file <none>
I20260928 02:05:07.782074 8466 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:07.782171 8466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:07.782428 8466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:07.785429 8466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/data/instance:
uuid: "6fb0862e5a314154968a6294cee9cc6a"
format_stamp: "Formatted at 2026-09-28 02:05:07 on dist-test-slave-23bp"
server_key: "addea5039bd073601a358764407b1712"
server_key_iv: "e69f618040a8803aef08691dd3684ac2"
server_key_version: "encryptionkey@0"
I20260928 02:05:07.786132 8466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/wal/instance:
uuid: "6fb0862e5a314154968a6294cee9cc6a"
format_stamp: "Formatted at 2026-09-28 02:05:07 on dist-test-slave-23bp"
server_key: "addea5039bd073601a358764407b1712"
server_key_iv: "e69f618040a8803aef08691dd3684ac2"
server_key_version: "encryptionkey@0"
I20260928 02:05:07.790923 8466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260928 02:05:07.794862 8566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:07.796259 8466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:05:07.796435 8466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/wal
uuid: "6fb0862e5a314154968a6294cee9cc6a"
format_stamp: "Formatted at 2026-09-28 02:05:07 on dist-test-slave-23bp"
server_key: "addea5039bd073601a358764407b1712"
server_key_iv: "e69f618040a8803aef08691dd3684ac2"
server_key_version: "encryptionkey@0"
I20260928 02:05:07.796725 8466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:07.808599 8466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:07.812438 8466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:07.812711 8466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:07.833186 8466 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.188:45051
I20260928 02:05:07.833209 8617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.188:45051 every 8 connection(s)
I20260928 02:05:07.834580 8466 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/data/info.pb
I20260928 02:05:07.837180 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 8466
I20260928 02:05:07.838654 8618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:07.848083 8618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:05:07 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.182.188@KRBTEST.COM
I20260928 02:05:07.866194 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.839353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49354 (local address 127.1.182.188:45051)
0928 02:05:07.840132 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:07.840143 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:07.840181 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:05:07.841723 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:07.841744 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:07.842140 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:07.842568 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:07.844781 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.846355 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.847714 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.848739 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.855252 (+ 6513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.855728 (+ 476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:07.855741 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:07.855764 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:07.855826 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:07.861254 (+ 5428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.862194 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.862203 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.862211 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.862396 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.862863 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.862867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.862870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.863633 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:07.863936 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:07.864588 (+ 652us) server_negotiation.cc:299] Negotiation successful
0928 02:05:07.865186 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":393,"threads_started":1}
I20260928 02:05:07.870581 8624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.848129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.1.182.189:39971)
0928 02:05:07.862978 (+ 14849us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:07.862984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:07.863952 (+ 968us) server_negotiation.cc:407] Connection header received
0928 02:05:07.864038 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:07.864043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:07.864144 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:07.864400 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:07.865003 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.866546 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.868500 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.869633 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.869789 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.869960 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:07.870260 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 02:05:07.870357 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14632,"thread_start_us":14527,"threads_started":1}
I20260928 02:05:07.870702 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.847887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39971 (local address 127.0.0.1:41870)
0928 02:05:07.863583 (+ 15696us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:07.863608 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:07.863838 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:07.864611 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:07.864616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:07.864641 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:07.864842 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.864850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.866691 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.866697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.868343 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.868354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.869851 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.869860 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.870060 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.870129 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 02:05:07.870234 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:05:07.870390 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15588,"thread_start_us":15504,"threads_started":1}
Sep 28 02:05:07 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561107, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.188@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM
I20260928 02:05:07.871925 8512 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260928 02:05:07.872908 8540 sys_catalog.cc:455] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bed972ae228a4bca907f46323a578168. Latest consensus state: current_term: 1 leader_uuid: "bed972ae228a4bca907f46323a578168" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } } }
I20260928 02:05:07.873577 8540 sys_catalog.cc:458] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:05:07.878731 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.851923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.1.182.190:39651)
0928 02:05:07.853744 (+ 1821us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:07.853753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:07.863172 (+ 9419us) server_negotiation.cc:407] Connection header received
0928 02:05:07.863456 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:07.863463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:07.863542 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:07.863614 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:07.865269 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.866360 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.868248 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.868723 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.868865 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.873341 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:07.873347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:07.873352 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:07.873394 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:07.876057 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.876771 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.876776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.876780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.876919 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.877290 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.877295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.877299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.877564 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:07.877716 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:07.878161 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 02:05:07.878373 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1648}
I20260928 02:05:07.878800 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.851294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.0.0.1:44954)
0928 02:05:07.861697 (+ 10403us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:07.861760 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:07.863311 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:07.863854 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:07.863861 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:07.864786 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:07.865117 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.865131 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.866525 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.866530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.868084 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.868097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.868909 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.868915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.869174 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.869961 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:07.869982 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:07.873163 (+ 3181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:07.876190 (+ 3027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.876204 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.876211 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.876643 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.877036 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.877041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.877045 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.877178 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.877703 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:07.877712 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:07.877879 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 02:05:07.878063 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 02:05:07.878380 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10121,"thread_start_us":10505,"threads_started":1}
I20260928 02:05:07.880119 8618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:05:07 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561107, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.188@KRBTEST.COM for oryx/127.1.182.189@KRBTEST.COM
I20260928 02:05:07.891026 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.880742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39971 (local address 127.0.0.1:41880)
0928 02:05:07.880920 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:07.880947 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:07.881028 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:07.881258 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:07.881261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:07.881543 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:07.881668 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.881675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.882590 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.882593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.883560 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.883579 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.884193 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.884196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.884332 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.884777 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:07.884805 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:07.886580 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:07.889200 (+ 2620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.889203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.889206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.889551 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.889850 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.889853 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.889855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.889970 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.890395 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:07.890398 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:07.890516 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 02:05:07.890619 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 02:05:07.890806 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 02:05:07.891198 8624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.880819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41880 (local address 127.1.182.189:39971)
0928 02:05:07.881020 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:07.881027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:07.881048 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:05:07.881094 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:07.881098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:07.881152 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:07.881229 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:07.881786 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.882482 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.883681 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.884078 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.884166 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.886741 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:07.886745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:07.886749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:07.886776 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:07.889091 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.889644 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.889648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.889650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.889760 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.890056 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.890059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.890061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.890304 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:07.890409 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:07.890698 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 02:05:07.890884 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 02:05:07.891978 8618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:05:07 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561107, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.188@KRBTEST.COM for oryx/127.1.182.188@KRBTEST.COM
I20260928 02:05:07.904167 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.892884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45051 (local address 127.0.0.1:49366)
0928 02:05:07.893093 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:07.893107 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:07.893188 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:07.893455 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:07.893461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:07.893748 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:07.893869 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.893876 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.894797 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.894800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.896081 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.896092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.896761 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.896765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.896896 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.897379 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:07.897395 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:07.899359 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:07.902064 (+ 2705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.902068 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.902071 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.902389 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.902692 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.902695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.902698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.902830 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.903331 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:07.903334 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:07.903455 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 02:05:07.903596 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 02:05:07.903792 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 02:05:07.904217 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.892878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49366 (local address 127.1.182.188:45051)
0928 02:05:07.893053 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:07.893063 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:07.893170 (+ 107us) server_negotiation.cc:407] Connection header received
0928 02:05:07.893271 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:07.893274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:07.893326 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:07.893408 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:07.893992 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.894693 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.896212 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.896621 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.896713 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.899511 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:07.899515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:07.899519 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:07.899545 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:07.901948 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.902488 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.902491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.902494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.902601 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.902917 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.902919 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.902922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.903232 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:07.903338 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:07.903692 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 02:05:07.903907 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":69}
I20260928 02:05:07.909068 8618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: Bootstrap starting.
I20260928 02:05:07.912693 8618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:07.914108 8618 log.cc:824] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:05:07.916851 8618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: No bootstrap required, opened a new log
I20260928 02:05:07.924324 8618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } }
I20260928 02:05:07.924651 8618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:07.924767 8618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb0862e5a314154968a6294cee9cc6a, State: Initialized, Role: FOLLOWER
I20260928 02:05:07.925325 8618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } }
I20260928 02:05:07.926302 8627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } } }
I20260928 02:05:07.926622 8627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:05:07.927353 8618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:05:07.936700 8638 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:05:07.936834 8638 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:05:07.940938 8466 master_runner.cc:428] Detected that this master 6fb0862e5a314154968a6294cee9cc6a is joining an existing cluster
I20260928 02:05:07.941099 8466 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.182.188:45051
I20260928 02:05:07.963356 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.946425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44960 (local address 127.1.182.190:39651)
0928 02:05:07.946606 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:07.946613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:07.947697 (+ 1084us) server_negotiation.cc:407] Connection header received
0928 02:05:07.947901 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:07.947914 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:07.947978 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:07.948083 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:07.949043 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.949771 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.951270 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.952054 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.952176 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.955600 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:07.955607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:07.955611 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:07.955652 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:07.959723 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.960499 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.960508 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.960513 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.960667 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.961229 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.961234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.961237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.961620 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:07.961746 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:07.962678 (+ 932us) server_negotiation.cc:299] Negotiation successful
0928 02:05:07.962912 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:05:07.965623 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.945758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.0.0.1:44960)
0928 02:05:07.947583 (+ 1825us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:07.947604 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:07.947765 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:07.948101 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:07.948105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:07.948619 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:07.948893 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.948901 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.949911 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.949918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.951044 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.951056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.952529 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.952540 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.952701 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.953605 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:07.953638 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:07.955228 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:07.959890 (+ 4662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.959896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.959900 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.960365 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.960805 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.960811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.960817 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.961070 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.961821 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:07.961826 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:07.962145 (+ 319us) client_negotiation.cc:769] Sending connection context
0928 02:05:07.963911 (+ 1766us) client_negotiation.cc:240] Negotiation successful
0928 02:05:07.964234 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1693,"thread_start_us":1535,"threads_started":1}
I20260928 02:05:07.975418 8624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.946214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41896 (local address 127.1.182.189:39971)
0928 02:05:07.946620 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:07.946627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:07.959415 (+ 12788us) server_negotiation.cc:407] Connection header received
0928 02:05:07.959714 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:07.959724 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:07.959836 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:07.959922 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:07.961918 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.963046 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.965128 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.965764 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.965876 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.968457 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:07.968466 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:07.968474 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:07.968512 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:07.972196 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.973027 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.973032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.973035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.973182 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.973638 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.973646 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.973651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.974008 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:07.974129 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:07.974766 (+ 637us) server_negotiation.cc:299] Negotiation successful
0928 02:05:07.975012 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305}
I20260928 02:05:07.976153 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 7964
I20260928 02:05:07.976509 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.945823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39971 (local address 127.0.0.1:41896)
0928 02:05:07.959281 (+ 13458us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:07.959309 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:07.959547 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:07.960183 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:07.960191 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:07.961370 (+ 1179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:07.961737 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.961747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.963278 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.963283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.964978 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.964990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.966193 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.966199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.966377 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.967056 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:07.967117 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:07.968262 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:07.972331 (+ 4069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.972337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.972341 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.972869 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.973334 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.973341 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.973346 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.973511 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.974227 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:07.974232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:07.974459 (+ 227us) client_negotiation.cc:769] Sending connection context
0928 02:05:07.975615 (+ 1156us) client_negotiation.cc:240] Negotiation successful
0928 02:05:07.975926 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9748,"thread_start_us":9211,"threads_started":1}
I20260928 02:05:07.983348 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.946242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49380 (local address 127.1.182.188:45051)
0928 02:05:07.946761 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:07.946765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:07.961362 (+ 14597us) server_negotiation.cc:407] Connection header received
0928 02:05:07.961537 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:07.961541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:07.961621 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:07.961726 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:07.963132 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.964246 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.967499 (+ 3253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.968202 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.968298 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.970711 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:07.970719 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:07.970725 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:07.970760 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:07.979616 (+ 8856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.980640 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.980648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.980654 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.980827 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.981287 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.981294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.981299 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.981752 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:07.981954 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:07.982526 (+ 572us) server_negotiation.cc:299] Negotiation successful
0928 02:05:07.982800 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406}
I20260928 02:05:07.984946 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.946004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45051 (local address 127.0.0.1:49380)
0928 02:05:07.961273 (+ 15269us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:07.961291 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:07.961398 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:07.961864 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:07.961869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:07.962656 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:07.962948 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.962958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.964530 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.964535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.967342 (+ 2807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.967356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.968590 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.968598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.968769 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.969291 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:07.969368 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:07.970532 (+ 1164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:07.979779 (+ 9247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.979787 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.979793 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.980496 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.980987 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.980994 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.981000 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.981158 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.982033 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:07.982040 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:07.982208 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 02:05:07.984209 (+ 2001us) client_negotiation.cc:240] Negotiation successful
0928 02:05:07.984483 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15145,"thread_start_us":15003,"threads_started":1}
Sep 28 02:05:07 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561104, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.190@KRBTEST.COM for oryx/127.1.182.188@KRBTEST.COM
I20260928 02:05:07.998581 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.971628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45051 (local address 127.0.0.1:49396)
0928 02:05:07.971810 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:07.971831 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:07.971931 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:07.984054 (+ 12123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:07.984078 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:07.984564 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:07.984770 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.984783 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.986347 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.986352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.987358 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:07.987371 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.988140 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.988146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:07.988341 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.989017 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:07.989039 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:07.991737 (+ 2698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:07.995731 (+ 3994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.995740 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.995747 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.996293 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.996698 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:07.996703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:07.996707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:07.996900 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:07.997501 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:07.997507 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:07.997694 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 02:05:07.997871 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 02:05:07.998169 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 02:05:07.998597 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.972141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49396 (local address 127.1.182.188:45051)
0928 02:05:07.983651 (+ 11510us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:07.983661 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:07.983683 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:05:07.983787 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:07.983795 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:07.983880 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:07.984007 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:07.984956 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.986192 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.987518 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.987984 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.988123 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:07.991929 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:07.991936 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:07.991941 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:07.991983 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:07.995597 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.996420 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.996427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.996436 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.996571 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:07.997028 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:07.997034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:07.997037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:07.997362 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:07.997507 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:07.997969 (+ 462us) server_negotiation.cc:299] Negotiation successful
0928 02:05:07.998224 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11349,"thread_start_us":11639,"threads_started":1}
Sep 28 02:05:07 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.182.129@KRBTEST.COM
I20260928 02:05:08.000070 8627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:08.000393 7902 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.182.188:45051: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a"
member_type: NON_VOTER
last_known_addr {
host: "127.1.182.188"
port: 45051
}
attrs {
promote: true
}
}
dest_uuid: "bed972ae228a4bca907f46323a578168"
cas_config_opid_index: 5
I20260928 02:05:08.000379 8627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } }
I20260928 02:05:08.001385 7902 consensus_queue.cc:237] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } }
I20260928 02:05:08.003667 7913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fb0862e5a314154968a6294cee9cc6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bed972ae228a4bca907f46323a578168" is_pre_election: true
I20260928 02:05:08.003460 8622 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6fb0862e5a314154968a6294cee9cc6a (127.1.182.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } } }
I20260928 02:05:08.003175 8627 leader_election.cc:288] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bed972ae228a4bca907f46323a578168 (127.1.182.190:39651), d64d03577e9843debd2679fd7e2b7ec6 (127.1.182.189:39971)
I20260928 02:05:08.007673 8512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fb0862e5a314154968a6294cee9cc6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d64d03577e9843debd2679fd7e2b7ec6" is_pre_election: true
I20260928 02:05:08.008086 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:07.977623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55364 (local address 127.1.182.129:46607)
0928 02:05:07.988640 (+ 11017us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:07.988648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:07.988674 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:05:07.988727 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:07.988733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:07.989055 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:07.989233 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:07.990857 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.992915 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.994351 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:07.994952 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:07.995123 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:08.000504 (+ 5381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:08.000518 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:08.000539 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:08.000643 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:08.005138 (+ 4495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:08.005768 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:08.005778 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:08.005781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:08.005954 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:08.006337 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:08.006343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:08.006346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:08.006679 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:08.006992 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:08.007393 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 02:05:08.007649 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10813,"thread_start_us":10642,"threads_started":1}
I20260928 02:05:08.008994 8512 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 1 LEARNER]: Refusing update from remote peer bed972ae228a4bca907f46323a578168: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 02:05:08.009486 8087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:08.009294 8568 leader_election.cc:302] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb0862e5a314154968a6294cee9cc6a; no voters: bed972ae228a4bca907f46323a578168, d64d03577e9843debd2679fd7e2b7ec6
I20260928 02:05:08.010468 8593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:08.011736 8649 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 02:05:08.017585 8593 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 FOLLOWER]: Refusing update from remote peer bed972ae228a4bca907f46323a578168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 02:05:08.017828 8627 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:05:08.018718 8622 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.182.188:45051
I20260928 02:05:08.019205 8622 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bed972ae228a4bca907f46323a578168" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } } }
I20260928 02:05:08.019531 8622 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:08.022782 8650 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.001s
I20260928 02:05:08.031533 8512 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6fb0862e5a314154968a6294cee9cc6a (127.1.182.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } } }
I20260928 02:05:08.034624 8540 sys_catalog.cc:455] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bed972ae228a4bca907f46323a578168" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } } }
I20260928 02:05:08.034890 8540 sys_catalog.cc:458] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:05:08.035655 8649 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168: attempting to promote NON_VOTER d64d03577e9843debd2679fd7e2b7ec6 to VOTER
I20260928 02:05:08.036293 8649 consensus_queue.cc:237] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } }
I20260928 02:05:08.038409 8512 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 1 LEARNER]: Refusing update from remote peer bed972ae228a4bca907f46323a578168: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 02:05:08.038515 8592 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 FOLLOWER]: Refusing update from remote peer bed972ae228a4bca907f46323a578168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 02:05:08.039175 8649 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:05:08.039469 8622 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 02:05:08.041828 8622 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d64d03577e9843debd2679fd7e2b7ec6 (127.1.182.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } } }
I20260928 02:05:08.042521 8512 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d64d03577e9843debd2679fd7e2b7ec6 (127.1.182.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } } }
I20260928 02:05:08.046285 8650 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bed972ae228a4bca907f46323a578168" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } } }
I20260928 02:05:08.046581 8650 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:08.046828 8540 sys_catalog.cc:455] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bed972ae228a4bca907f46323a578168" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } } }
I20260928 02:05:08.047027 8540 sys_catalog.cc:458] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:05:08.047596 8650 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bed972ae228a4bca907f46323a578168" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } } }
I20260928 02:05:08.047792 8650 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:08.051447 8466 master.cc:537] Master@127.1.182.188:45051 shutting down...
I20260928 02:05:08.062546 8593 raft_consensus.cc:858] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } }
W20260928 02:05:08.062755 8593 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } }
W20260928 02:05:08.064610 8572 tablet.cc:2369] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:05:08.079270 8593 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d64d03577e9843debd2679fd7e2b7ec6 (127.1.182.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } } }
I20260928 02:05:08.087471 8659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bed972ae228a4bca907f46323a578168" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } } }
I20260928 02:05:08.087996 8659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 02:05:08.087872 8656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bed972ae228a4bca907f46323a578168" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } } }
I20260928 02:05:08.088234 8656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 02:05:08.088363 8627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bed972ae228a4bca907f46323a578168. Latest consensus state: current_term: 1 leader_uuid: "bed972ae228a4bca907f46323a578168" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } } }
I20260928 02:05:08.088537 8627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 02:05:08.088809 8592 raft_consensus.cc:1467] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20260928 02:05:08.089087 8592 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 NON_PARTICIPANT]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.5->[1.6-1.7]
I20260928 02:05:08.089671 8592 raft_consensus.cc:858] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } }
W20260928 02:05:08.089768 8592 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 LEARNER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } }
I20260928 02:05:08.090176 8592 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6fb0862e5a314154968a6294cee9cc6a (127.1.182.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } } }
I20260928 02:05:08.093403 8627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bed972ae228a4bca907f46323a578168" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: NON_VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: true } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } } }
I20260928 02:05:08.093686 8627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: This master's current role is: LEARNER
I20260928 02:05:08.094789 8622 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bed972ae228a4bca907f46323a578168" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } } }
I20260928 02:05:08.095101 8622 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:08.096400 8466 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 LEARNER]: Raft consensus shutting down.
I20260928 02:05:08.097014 8466 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 LEARNER]: Raft consensus is shut down!
I20260928 02:05:08.097240 8466 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: stopping tablet replica
W20260928 02:05:08.106365 7890 consensus_peers.cc:487] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 -> Peer 6fb0862e5a314154968a6294cee9cc6a (127.1.182.188:45051): Couldn't send request to peer 6fb0862e5a314154968a6294cee9cc6a. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:05:08.117156 8466 master.cc:552] Master@127.1.182.188:45051 shutdown complete.
I20260928 02:05:08.117257 8466 master_runner.cc:305] Clearing existing system tablet
I20260928 02:05:08.117601 8466 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:08.117713 8466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:08.119918 8466 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:05:08.121342 8662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:08.121737 8466 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:08.121978 8466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:05:08.122077 8466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/wal
uuid: "6fb0862e5a314154968a6294cee9cc6a"
format_stamp: "Formatted at 2026-09-28 02:05:07 on dist-test-slave-23bp"
server_key: "addea5039bd073601a358764407b1712"
server_key_iv: "e69f618040a8803aef08691dd3684ac2"
server_key_version: "encryptionkey@0"
I20260928 02:05:08.125222 8466 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:08.130219 8466 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:05:08.130371 8466 log.cc:1196] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 02:05:08.130872 8466 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: Deleting consensus metadata
I20260928 02:05:08.131145 8466 master_runner.cc:315] Copying system tablet from 127.1.182.190:39651
I20260928 02:05:08.134042 8466 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: tablet copy: Beginning tablet copy session from remote peer at address 127.1.182.190:39651
I20260928 02:05:08.147550 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:08.134799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.1.182.190:39651)
0928 02:05:08.135105 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:08.135115 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:08.135644 (+ 529us) server_negotiation.cc:407] Connection header received
0928 02:05:08.135865 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:08.135871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:08.135948 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:08.136051 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:08.136837 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.137526 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.138806 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.139473 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.139566 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:08.141993 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:08.142000 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:08.142005 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:08.142042 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:08.144992 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:08.145608 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:08.145615 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:08.145618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:08.145746 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:08.146126 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:08.146132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:08.146136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:08.146398 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:08.146555 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:08.146943 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 02:05:08.147187 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:05:08.147915 8668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:08.134776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.0.0.1:44976)
0928 02:05:08.135526 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:08.135546 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:08.135717 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:08.136078 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:08.136085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:08.136500 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:08.136688 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:08.136698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.137660 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.137666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:08.138642 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:08.138657 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.139799 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.139804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:08.140005 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:08.140712 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:08.140735 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:08.141736 (+ 1001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:08.145124 (+ 3388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:08.145131 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:08.145136 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:08.145492 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:08.145856 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:08.145861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:08.145865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:08.146019 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:08.146539 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:08.146544 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:08.146703 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 02:05:08.146850 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 02:05:08.147186 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":477,"threads_started":1}
I20260928 02:05:08.148563 7923 tablet_copy_service.cc:140] P bed972ae228a4bca907f46323a578168: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6fb0862e5a314154968a6294cee9cc6a ({username='oryx', principal='oryx/127.1.182.188@KRBTEST.COM'} at 127.0.0.1:44976)
I20260928 02:05:08.148723 7923 tablet_copy_service.cc:161] P bed972ae228a4bca907f46323a578168: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6fb0862e5a314154968a6294cee9cc6a at {username='oryx', principal='oryx/127.1.182.188@KRBTEST.COM'} at 127.0.0.1:44976: session id = 6fb0862e5a314154968a6294cee9cc6a-00000000000000000000000000000000
I20260928 02:05:08.151017 7923 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:05:08.152629 8466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:05:08.157694 8466 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: tablet copy: Starting download of 0 data blocks...
I20260928 02:05:08.157963 8466 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: tablet copy: Starting download of 1 WAL segments...
I20260928 02:05:08.160069 8466 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:05:08.164275 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 7964
I20260928 02:05:08.165004 7923 tablet_copy_service.cc:342] P bed972ae228a4bca907f46323a578168: Request end of tablet copy session 6fb0862e5a314154968a6294cee9cc6a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.182.188@KRBTEST.COM'} at 127.0.0.1:44976
I20260928 02:05:08.165158 7923 tablet_copy_service.cc:433] P bed972ae228a4bca907f46323a578168: ending tablet copy session 6fb0862e5a314154968a6294cee9cc6a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6fb0862e5a314154968a6294cee9cc6a
I20260928 02:05:08.182541 8466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:05:08.184688 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:46607
--local_ip_for_outbound_sockets=127.1.182.129
--tserver_master_addrs=127.1.182.190:39651
--webserver_port=32911
--webserver_interface=127.1.182.129
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.129.keytab
--principal=oryx/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.182.190:39651,127.1.182.189:39971
--tserver_master_addrs=127.1.182.190:39651,127.1.182.189:39971,127.1.182.188:45051 with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:08.185794 8669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:08.199007 8673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:08.199272 8466 server_base.cc:1034] running on GCE node
W20260928 02:05:08.199563 8670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:08.200066 8466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:08.200822 8466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:08.202064 8466 hybrid_clock.cc:648] HybridClock initialized: now 1790561108202008 us; error 54 us; skew 500 ppm
I20260928 02:05:08.202370 8466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:08 dist-test-slave-23bp krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561108, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:08.204957 8466 init.cc:377] Logged in from keytab as oryx/127.1.182.188@KRBTEST.COM (short username oryx)
I20260928 02:05:08.208006 8466 webserver.cc:460] Webserver started at http://127.1.182.188:35733/ using document root <none> and password file <none>
I20260928 02:05:08.208384 8466 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:08.208575 8466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:08.211705 8466 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:05:08.213660 8680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:08.214073 8466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:05:08.214279 8466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/wal
uuid: "6fb0862e5a314154968a6294cee9cc6a"
format_stamp: "Formatted at 2026-09-28 02:05:07 on dist-test-slave-23bp"
server_key: "addea5039bd073601a358764407b1712"
server_key_iv: "e69f618040a8803aef08691dd3684ac2"
server_key_version: "encryptionkey@0"
I20260928 02:05:08.214540 8466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:08.234465 8466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:08.238634 8466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:08.273063 8466 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.188:45051
I20260928 02:05:08.273123 8731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.188:45051 every 8 connection(s)
I20260928 02:05:08.274227 8466 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/master-2/data/info.pb
I20260928 02:05:08.279762 8732 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:05:08.283886 8732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: Bootstrap starting.
I20260928 02:05:08.298571 8732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:05:08.299201 8732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: Bootstrap complete.
I20260928 02:05:08.300485 8732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } }
I20260928 02:05:08.300774 8732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6fb0862e5a314154968a6294cee9cc6a, State: Initialized, Role: LEARNER
I20260928 02:05:08.301039 8732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } }
I20260928 02:05:08.301857 8735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } } }
I20260928 02:05:08.302143 8732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:05:08.302400 8735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: This master's current role is: LEARNER
I20260928 02:05:08.316493 8746 catalog_manager.cc:1259] Loaded cluster ID: 341136e4888648f094f3e854341582e0
I20260928 02:05:08.316632 8746 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: loading cluster ID for follower catalog manager: success
I20260928 02:05:08.320567 8746 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: acquiring CA information for follower catalog manager: success
I20260928 02:05:08.323012 8746 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 02:05:08.549206 8671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:08.549487 8671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:08.549588 8671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:08.557256 8671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:08.557366 8671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:08.557476 8671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:05:08.567349 8671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.129.keytab
--principal=oryx/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:46607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=32911
--tserver_master_addrs=127.1.182.190:39651,127.1.182.189:39971,127.1.182.188:45051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:08.568595 8671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:08.570343 8671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:08.583709 8754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:08.583845 8751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:08.583788 8752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:08.584689 8671 server_base.cc:1034] running on GCE node
I20260928 02:05:08.585233 8671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:08.586090 8671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:08.587283 8671 hybrid_clock.cc:648] HybridClock initialized: now 1790561108587234 us; error 44 us; skew 500 ppm
I20260928 02:05:08.587554 8671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:08 dist-test-slave-23bp krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561108, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:08.590500 8671 init.cc:377] Logged in from keytab as oryx/127.1.182.129@KRBTEST.COM (short username oryx)
I20260928 02:05:08.591964 8671 webserver.cc:460] Webserver started at http://127.1.182.129:32911/ using document root <none> and password file <none>
I20260928 02:05:08.592662 8671 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:08.592809 8671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:08.598304 8671 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:05:08.601487 8761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:08.603349 8671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:05:08.603521 8671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "9d156a26f9e140dcb64782bed8729d55"
format_stamp: "Formatted at 2026-09-28 02:05:05 on dist-test-slave-23bp"
server_key: "0b803846949a275626f90b324b7cd9d5"
server_key_iv: "4ab0a075b676ba1b78928c3dc87d10f7"
server_key_version: "encryptionkey@0"
I20260928 02:05:08.604489 8671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:08.630472 8671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:08.632721 8766 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bed972ae228a4bca907f46323a578168" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } } }
I20260928 02:05:08.633184 8766 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:08.641856 8671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:08.642133 8671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:08.643268 8671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:08.645215 8671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:08.645334 8671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:08.645464 8671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:08.645576 8671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:08.646054 8769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:08.632948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41314 (local address 127.1.182.188:45051)
0928 02:05:08.640963 (+ 8015us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:08.640968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:08.640983 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:05:08.641042 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:08.641045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:08.641117 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:08.641250 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:08.641680 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.642721 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.644328 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.645096 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.645254 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:08.645494 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:08.645627 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 02:05:08.645751 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7882,"thread_start_us":7772,"threads_started":1}
I20260928 02:05:08.646056 8770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:08.633058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45051 (local address 127.0.0.1:41314)
0928 02:05:08.640568 (+ 7510us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:08.640589 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:08.640740 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:08.641272 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:08.641277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:08.641304 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:08.641518 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:08.641529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.642870 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.642876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:08.644174 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:08.644186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.645276 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.645282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:08.645454 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:08.645492 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:05:08.645587 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 02:05:08.645722 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7397,"thread_start_us":7347,"threads_started":1}
I20260928 02:05:08.646937 8707 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 02:05:08.647356 8735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bed972ae228a4bca907f46323a578168. Latest consensus state: current_term: 1 leader_uuid: "bed972ae228a4bca907f46323a578168" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: NON_VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: true } } }
I20260928 02:05:08.647640 8735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: This master's current role is: LEARNER
I20260928 02:05:08.648293 8767 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168: attempting to promote NON_VOTER 6fb0862e5a314154968a6294cee9cc6a to VOTER
I20260928 02:05:08.648859 8767 consensus_queue.cc:237] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: false } }
I20260928 02:05:08.651203 8512 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 1 FOLLOWER]: Refusing update from remote peer bed972ae228a4bca907f46323a578168: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 02:05:08.651305 8707 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 LEARNER]: Refusing update from remote peer bed972ae228a4bca907f46323a578168: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 02:05:08.651928 8768 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 02:05:08.652238 8767 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 02:05:08.655007 8767 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6fb0862e5a314154968a6294cee9cc6a (127.1.182.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: false } } }
I20260928 02:05:08.655777 8512 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6fb0862e5a314154968a6294cee9cc6a (127.1.182.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: false } } }
I20260928 02:05:08.658084 8766 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bed972ae228a4bca907f46323a578168" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: false } } }
I20260928 02:05:08.658325 8766 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:08.658110 8767 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bed972ae228a4bca907f46323a578168" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: false } } }
I20260928 02:05:08.658428 8767 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:08.660756 8766 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bed972ae228a4bca907f46323a578168" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: false } } }
I20260928 02:05:08.660979 8766 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:08.661303 8707 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6fb0862e5a314154968a6294cee9cc6a (127.1.182.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: false } } }
I20260928 02:05:08.664480 8735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bed972ae228a4bca907f46323a578168" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: false } } }
I20260928 02:05:08.664723 8735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:05:08.667477 8785 sys_catalog.cc:455] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bed972ae228a4bca907f46323a578168" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: false } } }
I20260928 02:05:08.667857 8785 sys_catalog.cc:458] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:05:08.701287 8671 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:46607
I20260928 02:05:08.701386 8881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:46607 every 8 connection(s)
I20260928 02:05:08.702982 8671 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:05:08.711647 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 8671
I20260928 02:05:08.712046 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 8098
Sep 28 02:05:08 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561108, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM
Sep 28 02:05:08 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561108, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.188@KRBTEST.COM
Sep 28 02:05:08 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561108, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.189@KRBTEST.COM
Sep 28 02:05:08 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.182.130@KRBTEST.COM
I20260928 02:05:08.758540 8769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:08.706830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:39071 (local address 127.1.182.188:45051)
0928 02:05:08.707119 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:08.707128 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:08.731703 (+ 24575us) server_negotiation.cc:407] Connection header received
0928 02:05:08.736241 (+ 4538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:08.736246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:08.736336 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:08.736407 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:08.739470 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.740557 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.743593 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.744260 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.744356 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:08.750669 (+ 6313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:08.750676 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:08.750680 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:08.750731 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:08.754607 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:08.755728 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:08.755735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:08.755738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:08.755866 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:08.756398 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:08.756405 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:08.756411 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:08.756779 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:08.756902 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:08.757949 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0928 02:05:08.758175 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:05:08.758574 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:08.706190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:54377 (local address 127.1.182.189:39971)
0928 02:05:08.722733 (+ 16543us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:08.722742 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:08.733255 (+ 10513us) server_negotiation.cc:407] Connection header received
0928 02:05:08.735214 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:08.735221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:08.735322 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:08.735429 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:08.739475 (+ 4046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.740591 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.743593 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.744260 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.744365 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:08.751468 (+ 7103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:08.751494 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:08.751500 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:08.751545 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:08.755394 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:08.756204 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:08.756209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:08.756212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:08.756352 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:08.756760 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:08.756765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:08.756768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:08.757108 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:08.757222 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:08.757949 (+ 727us) server_negotiation.cc:299] Negotiation successful
0928 02:05:08.758174 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16372,"thread_start_us":16218,"threads_started":1}
I20260928 02:05:08.759788 8887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:08.706532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:41371 (local address 127.1.182.190:39651)
0928 02:05:08.736631 (+ 30099us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:08.736637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:08.736654 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:05:08.736707 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:08.736713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:08.736808 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:08.737025 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:08.739469 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.741049 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.743646 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.744270 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.744405 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:08.751132 (+ 6727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:08.751139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:08.751144 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:08.751185 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:08.754938 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:08.755904 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:08.755910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:08.755915 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:08.756072 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:08.756523 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:08.756528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:08.756531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:08.756877 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:08.757102 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:08.759226 (+ 2124us) server_negotiation.cc:299] Negotiation successful
0928 02:05:08.759425 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29951,"thread_start_us":29811,"threads_started":1}
I20260928 02:05:08.759883 8892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:08.713149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57274 (local address 127.1.182.130:39733)
0928 02:05:08.730584 (+ 17435us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:08.730598 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:08.736008 (+ 5410us) server_negotiation.cc:407] Connection header received
0928 02:05:08.736177 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:08.736190 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:08.736463 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:08.736623 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:08.738709 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.740803 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.742711 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.743563 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.743654 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:08.752228 (+ 8574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:08.752243 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:08.752263 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:08.752312 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:08.756539 (+ 4227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:08.757413 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:08.757423 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:08.757435 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:08.757592 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:08.758023 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:08.758028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:08.758031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:08.758418 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:08.758694 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:08.759282 (+ 588us) server_negotiation.cc:299] Negotiation successful
0928 02:05:08.759536 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17138,"thread_start_us":17454,"threads_started":1}
I20260928 02:05:08.760432 8889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:08.707329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39971 (local address 127.1.182.129:54377)
0928 02:05:08.733162 (+ 25833us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:08.733184 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:08.734836 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:08.736864 (+ 2028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:08.736868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:08.738032 (+ 1164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:08.739262 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:08.739288 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.740765 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.740769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:08.743427 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:08.743443 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.744730 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.744736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:08.745138 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:08.746096 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:08.746137 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:08.751217 (+ 5080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:08.755574 (+ 4357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:08.755580 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:08.755584 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:08.756073 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:08.756503 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:08.756508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:08.756511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:08.756640 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:08.757290 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:08.757294 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:08.757495 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 02:05:08.758799 (+ 1304us) client_negotiation.cc:240] Negotiation successful
0928 02:05:08.759126 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25728,"mutex_wait_us":483,"spinlock_wait_cycles":42496,"thread_start_us":27748,"threads_started":1}
I20260928 02:05:08.760464 8890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:08.706925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.129:41371)
0928 02:05:08.732599 (+ 25674us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:08.732622 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:08.734837 (+ 2215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:08.737247 (+ 2410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:08.737257 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:08.738052 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:08.739262 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:08.739289 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.741206 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.741211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:08.743477 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:08.743492 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.744720 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.744729 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:08.745149 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:08.746191 (+ 1042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:08.746218 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:08.750898 (+ 4680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:08.755113 (+ 4215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:08.755118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:08.755122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:08.755705 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:08.756209 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:08.756213 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:08.756218 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:08.756389 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:08.757164 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:08.757170 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:08.757551 (+ 381us) client_negotiation.cc:769] Sending connection context
0928 02:05:08.757829 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 02:05:08.758374 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25551,"mutex_wait_us":368,"spinlock_wait_cycles":33152,"thread_start_us":28927,"threads_started":1}
I20260928 02:05:08.761685 8221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:08.762064 8883 heartbeater.cc:347] Connected to a master server at 127.1.182.190:39651
I20260928 02:05:08.762383 8883 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:08.763195 8884 heartbeater.cc:347] Connected to a master server at 127.1.182.189:39971
I20260928 02:05:08.763247 8883 heartbeater.cc:511] Master 127.1.182.190:39651 requested a full tablet report, sending...
I20260928 02:05:08.763336 8884 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:08.763844 8884 heartbeater.cc:511] Master 127.1.182.189:39971 requested a full tablet report, sending...
I20260928 02:05:08.764380 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:08.706840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45051 (local address 127.1.182.129:39071)
0928 02:05:08.731510 (+ 24670us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:08.731575 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:08.735353 (+ 3778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:08.736721 (+ 1368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:08.736735 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:08.738032 (+ 1297us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:08.739267 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:08.739288 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.740762 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.740767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:08.743448 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:08.743457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:08.744718 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:08.744725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:08.745126 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:08.746096 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:08.746133 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:08.749974 (+ 3841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:08.754773 (+ 4799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:08.754784 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:08.754804 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:08.755596 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:08.756029 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:08.756034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:08.756037 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:08.756274 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:08.757001 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:08.757008 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:08.757495 (+ 487us) client_negotiation.cc:769] Sending connection context
0928 02:05:08.758799 (+ 1304us) client_negotiation.cc:240] Negotiation successful
0928 02:05:08.763622 (+ 4823us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24322,"mutex_wait_us":4804,"spinlock_wait_cycles":74112,"thread_start_us":24180,"threads_started":1}
I20260928 02:05:08.765107 7902 ts_manager.cc:194] Re-registered known tserver with Master: 9d156a26f9e140dcb64782bed8729d55 (127.1.182.129:46607)
I20260928 02:05:08.765635 8882 heartbeater.cc:347] Connected to a master server at 127.1.182.188:45051
I20260928 02:05:08.765630 8502 ts_manager.cc:194] Registered new tserver with Master: 9d156a26f9e140dcb64782bed8729d55 (127.1.182.129:46607)
I20260928 02:05:08.765784 8882 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:08.766227 8882 heartbeater.cc:511] Master 127.1.182.188:45051 requested a full tablet report, sending...
I20260928 02:05:08.766212 7902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:41371
I20260928 02:05:08.768489 8697 ts_manager.cc:194] Registered new tserver with Master: 9d156a26f9e140dcb64782bed8729d55 (127.1.182.129:46607)
I20260928 02:05:08.851891 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 8098
I20260928 02:05:08.869576 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:39733
--local_ip_for_outbound_sockets=127.1.182.130
--tserver_master_addrs=127.1.182.190:39651
--webserver_port=41013
--webserver_interface=127.1.182.130
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.130.keytab
--principal=oryx/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.182.190:39651,127.1.182.189:39971
--tserver_master_addrs=127.1.182.190:39651,127.1.182.189:39971,127.1.182.188:45051 with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:09.193650 8894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:09.193915 8894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:09.194016 8894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:09.201098 8894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:09.201197 8894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:09.201306 8894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:05:09.211082 8894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.130.keytab
--principal=oryx/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:39733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=41013
--tserver_master_addrs=127.1.182.190:39651,127.1.182.189:39971,127.1.182.188:45051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:09.212272 8894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:09.213549 8894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:09.227371 8899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:09.227402 8902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:09.227304 8900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:09.228300 8894 server_base.cc:1034] running on GCE node
I20260928 02:05:09.228919 8894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:09.229864 8894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:09.231057 8894 hybrid_clock.cc:648] HybridClock initialized: now 1790561109231018 us; error 40 us; skew 500 ppm
I20260928 02:05:09.231431 8894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:09 dist-test-slave-23bp krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561109, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:09.234452 8894 init.cc:377] Logged in from keytab as oryx/127.1.182.130@KRBTEST.COM (short username oryx)
I20260928 02:05:09.236047 8894 webserver.cc:460] Webserver started at http://127.1.182.130:41013/ using document root <none> and password file <none>
I20260928 02:05:09.236722 8894 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:09.236894 8894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:09.242201 8894 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:05:09.244889 8909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:09.246218 8894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:05:09.246383 8894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "5e5d0dbc247b46bc8219f4db2a911ec7"
format_stamp: "Formatted at 2026-09-28 02:05:06 on dist-test-slave-23bp"
server_key: "9f30befbdef3e81c42e675297672ff21"
server_key_iv: "4716a00e2827c6920732d381e725e390"
server_key_version: "encryptionkey@0"
I20260928 02:05:09.247404 8894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:09.275991 8894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:09.280114 8894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:09.280386 8894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:09.281337 8894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:09.282641 8894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:09.282764 8894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:09.282889 8894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:09.282980 8894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:09.315827 8894 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:39733
I20260928 02:05:09.315912 9021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:39733 every 8 connection(s)
I20260928 02:05:09.317260 8894 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:05:09.320370 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 8894
I20260928 02:05:09.320667 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 8232
Sep 28 02:05:09 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.182.131@KRBTEST.COM
Sep 28 02:05:09 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561109, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.188@KRBTEST.COM
Sep 28 02:05:09 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561109, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM
Sep 28 02:05:09 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561109, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.189@KRBTEST.COM
I20260928 02:05:09.372653 9030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:09.321606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:58245 (local address 127.1.182.188:45051)
0928 02:05:09.346435 (+ 24829us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:09.346445 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:09.347366 (+ 921us) server_negotiation.cc:407] Connection header received
0928 02:05:09.350612 (+ 3246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:09.350619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:09.350713 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:09.350792 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:09.354198 (+ 3406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.355318 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.358576 (+ 3258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.359224 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.359317 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:09.364571 (+ 5254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:09.364579 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:09.364584 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:09.364626 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:09.368518 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:09.369843 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:09.369850 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:09.369856 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:09.370010 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:09.370537 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:09.370544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:09.370550 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:09.370855 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:09.371009 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:09.372016 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0928 02:05:09.372235 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24655,"thread_start_us":24837,"threads_started":1}
I20260928 02:05:09.372669 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:09.322029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.1.182.131:33133)
0928 02:05:09.344489 (+ 22460us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:09.344502 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:09.345594 (+ 1092us) server_negotiation.cc:407] Connection header received
0928 02:05:09.345687 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:09.345700 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:09.346017 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:09.346207 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:09.349966 (+ 3759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.351699 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.353730 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.354981 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.355125 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:09.361484 (+ 6359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:09.361498 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:09.361519 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:09.361570 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:09.365718 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:09.368681 (+ 2963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:09.368691 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:09.368695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:09.368842 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:09.369311 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:09.369316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:09.369318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:09.369791 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:09.370092 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:09.371971 (+ 1879us) server_negotiation.cc:299] Negotiation successful
0928 02:05:09.372244 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22175,"thread_start_us":22475,"threads_started":1}
I20260928 02:05:09.374116 9026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:09.320858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45051 (local address 127.1.182.130:58245)
0928 02:05:09.347122 (+ 26264us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:09.347224 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:09.350452 (+ 3228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:09.351297 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:09.351309 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:09.352881 (+ 1572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:09.353989 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:09.354010 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.355519 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.355524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:09.358385 (+ 2861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:09.358394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.359665 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.359673 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:09.360608 (+ 935us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:09.361412 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:09.361456 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:09.364277 (+ 2821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:09.369022 (+ 4745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:09.369033 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:09.369055 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:09.369694 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:09.370165 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:09.370169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:09.370172 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:09.370417 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:09.370999 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:09.371007 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:09.371497 (+ 490us) client_negotiation.cc:769] Sending connection context
0928 02:05:09.373381 (+ 1884us) client_negotiation.cc:240] Negotiation successful
0928 02:05:09.373595 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25929,"mutex_wait_us":1167,"spinlock_wait_cycles":73472,"thread_start_us":25662,"threads_started":1}
I20260928 02:05:09.374189 8355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:09.374135 9027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:09.320859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39971 (local address 127.1.182.130:41811)
0928 02:05:09.347122 (+ 26263us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:09.347424 (+ 302us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:09.349997 (+ 2573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:09.351319 (+ 1322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:09.351322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:09.352881 (+ 1559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:09.353983 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:09.354010 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.355507 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.355512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:09.358366 (+ 2854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:09.358382 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.359665 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.359673 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:09.360607 (+ 934us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:09.362397 (+ 1790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:09.362420 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:09.366256 (+ 3836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:09.369989 (+ 3733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:09.369994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:09.369997 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:09.370327 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:09.370764 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:09.370772 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:09.370777 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:09.370992 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:09.371754 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:09.371758 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:09.371922 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 02:05:09.372173 (+ 251us) client_negotiation.cc:240] Negotiation successful
0928 02:05:09.372669 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26064,"mutex_wait_us":1236,"spinlock_wait_cycles":35328,"thread_start_us":25743,"threads_started":1}
I20260928 02:05:09.374792 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:09.321388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:41811 (local address 127.1.182.189:39971)
0928 02:05:09.350855 (+ 29467us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:09.350862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:09.350879 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:05:09.350950 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:09.350954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:09.351035 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:09.351160 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:09.354195 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.355318 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.358538 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.359218 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.359317 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:09.366489 (+ 7172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:09.366495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:09.366499 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:09.366545 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:09.369851 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:09.370453 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:09.370458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:09.370461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:09.370632 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:09.371334 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:09.371340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:09.371343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:09.371624 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:09.371772 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:09.374151 (+ 2379us) server_negotiation.cc:299] Negotiation successful
0928 02:05:09.374400 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29348,"thread_start_us":29473,"threads_started":1}
I20260928 02:05:09.376132 9022 heartbeater.cc:347] Connected to a master server at 127.1.182.188:45051
I20260928 02:05:09.376169 9024 heartbeater.cc:347] Connected to a master server at 127.1.182.189:39971
I20260928 02:05:09.376461 9022 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:09.376466 9024 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:09.376794 9032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:09.321936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:58727 (local address 127.1.182.190:39651)
0928 02:05:09.345919 (+ 23983us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:09.345938 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:09.350387 (+ 4449us) server_negotiation.cc:407] Connection header received
0928 02:05:09.350541 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:09.350547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:09.350642 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:09.350758 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:09.354192 (+ 3434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.355288 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.358575 (+ 3287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.359313 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.359414 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:09.365154 (+ 5740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:09.365161 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:09.365166 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:09.365220 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:09.369715 (+ 4495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:09.371938 (+ 2223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:09.371944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:09.371947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:09.372133 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:09.374598 (+ 2465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:09.374603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:09.374607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:09.374955 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:09.375128 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:09.376126 (+ 998us) server_negotiation.cc:299] Negotiation successful
0928 02:05:09.376382 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23807,"thread_start_us":24002,"threads_started":1}
I20260928 02:05:09.377540 9024 heartbeater.cc:511] Master 127.1.182.189:39971 requested a full tablet report, sending...
I20260928 02:05:09.378350 9022 heartbeater.cc:511] Master 127.1.182.188:45051 requested a full tablet report, sending...
I20260928 02:05:09.379177 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:09.321733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.130:58727)
0928 02:05:09.350302 (+ 28569us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:09.350318 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:09.350437 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:09.351311 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:09.351316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:09.352866 (+ 1550us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:09.354004 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:09.354020 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.355508 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.355515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:09.358396 (+ 2881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:09.358415 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.359725 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.359734 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:09.360034 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:09.361545 (+ 1511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:09.361563 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:09.364919 (+ 3356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:09.371316 (+ 6397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:09.371320 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:09.371323 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:09.371804 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:09.372918 (+ 1114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:09.372923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:09.372926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:09.373091 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:09.375688 (+ 2597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:09.375693 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:09.375831 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 02:05:09.375996 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 02:05:09.376272 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28441,"mutex_wait_us":250,"spinlock_wait_cycles":45184,"thread_start_us":28321,"threads_started":1}
I20260928 02:05:09.379542 8697 ts_manager.cc:194] Registered new tserver with Master: 5e5d0dbc247b46bc8219f4db2a911ec7 (127.1.182.130:39733)
I20260928 02:05:09.379834 8502 ts_manager.cc:194] Registered new tserver with Master: 5e5d0dbc247b46bc8219f4db2a911ec7 (127.1.182.130:39733)
I20260928 02:05:09.380012 9023 heartbeater.cc:347] Connected to a master server at 127.1.182.190:39651
I20260928 02:05:09.380121 9023 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:09.380599 9023 heartbeater.cc:511] Master 127.1.182.190:39651 requested a full tablet report, sending...
I20260928 02:05:09.381768 7902 ts_manager.cc:194] Re-registered known tserver with Master: 5e5d0dbc247b46bc8219f4db2a911ec7 (127.1.182.130:39733)
I20260928 02:05:09.382899 7902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:58727
I20260928 02:05:09.461131 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 8232
I20260928 02:05:09.477505 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:33133
--local_ip_for_outbound_sockets=127.1.182.131
--tserver_master_addrs=127.1.182.190:39651
--webserver_port=44767
--webserver_interface=127.1.182.131
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.131.keytab
--principal=oryx/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.182.190:39651,127.1.182.189:39971
--tserver_master_addrs=127.1.182.190:39651,127.1.182.189:39971,127.1.182.188:45051 with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:05:09.769150 8883 heartbeater.cc:503] Master 127.1.182.190:39651 was elected leader, sending a full tablet report...
W20260928 02:05:09.802544 9035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:09.802830 9035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:09.803025 9035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:09.810745 9035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:09.810880 9035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:09.811041 9035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:05:09.820911 9035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.131.keytab
--principal=oryx/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:33133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=44767
--tserver_master_addrs=127.1.182.190:39651,127.1.182.189:39971,127.1.182.188:45051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:09.822216 9035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:09.823549 9035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:09.836601 9042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:09.836635 9044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:09.836601 9041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:09.837546 9035 server_base.cc:1034] running on GCE node
I20260928 02:05:09.838106 9035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:09.839039 9035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:09.840302 9035 hybrid_clock.cc:648] HybridClock initialized: now 1790561109840248 us; error 51 us; skew 500 ppm
I20260928 02:05:09.840610 9035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:09 dist-test-slave-23bp krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561109, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:09.843694 9035 init.cc:377] Logged in from keytab as oryx/127.1.182.131@KRBTEST.COM (short username oryx)
I20260928 02:05:09.845105 9035 webserver.cc:460] Webserver started at http://127.1.182.131:44767/ using document root <none> and password file <none>
I20260928 02:05:09.845736 9035 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:09.845842 9035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:09.851255 9035 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:05:09.853849 9051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:09.855242 9035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:05:09.855434 9035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "5db2590c7c594cd9b59e518991ff1445"
format_stamp: "Formatted at 2026-09-28 02:05:06 on dist-test-slave-23bp"
server_key: "0610761f5aab04d393478b83bec709b2"
server_key_iv: "2a6567e45532a2b66ba3ce77cc041a5d"
server_key_version: "encryptionkey@0"
I20260928 02:05:09.856501 9035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:09.873101 9035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:09.877384 9035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:09.877652 9035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:09.878768 9035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:09.880302 9035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:09.880419 9035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:09.880550 9035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:09.880637 9035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:09.913174 9035 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:33133
I20260928 02:05:09.913265 9163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:33133 every 8 connection(s)
I20260928 02:05:09.914896 9035 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:05:09.918391 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 9035
Sep 28 02:05:09 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561109, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.189@KRBTEST.COM
Sep 28 02:05:09 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561109, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.188@KRBTEST.COM
Sep 28 02:05:09 dist-test-slave-23bp krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561109, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM
I20260928 02:05:09.967655 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:09.918590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39971 (local address 127.1.182.131:58833)
0928 02:05:09.940272 (+ 21682us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:09.940287 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:09.941015 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:09.941790 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:09.941808 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:09.943162 (+ 1354us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:09.944295 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:09.944329 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.945880 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.945889 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:09.948663 (+ 2774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:09.948684 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.949735 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.949742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:09.950022 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:09.951084 (+ 1062us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:09.951110 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:09.955053 (+ 3943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:09.963865 (+ 8812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:09.963870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:09.963873 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:09.964799 (+ 926us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:09.965260 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:09.965264 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:09.965267 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:09.965464 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:09.966227 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:09.966235 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:09.966440 (+ 205us) client_negotiation.cc:769] Sending connection context
0928 02:05:09.966613 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 02:05:09.966951 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21544,"mutex_wait_us":1056,"spinlock_wait_cycles":54016,"thread_start_us":15876,"threads_started":1}
I20260928 02:05:09.968686 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:09.920529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:53743 (local address 127.1.182.188:45051)
0928 02:05:09.942739 (+ 22210us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:09.942745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:09.942764 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:05:09.942825 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:09.942834 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:09.942913 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:09.944046 (+ 1133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:09.945863 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.946941 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.948935 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.949438 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.949578 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:09.956554 (+ 6976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:09.956559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:09.956563 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:09.956604 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:09.960955 (+ 4351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:09.962051 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:09.962060 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:09.962065 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:09.962236 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:09.962841 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:09.962849 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:09.962854 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:09.963351 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:09.963544 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:09.967770 (+ 4226us) server_negotiation.cc:299] Negotiation successful
0928 02:05:09.968149 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22076,"thread_start_us":21923,"threads_started":1}
I20260928 02:05:09.969925 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:09.918855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:58833 (local address 127.1.182.189:39971)
0928 02:05:09.931404 (+ 12549us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:09.931414 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:09.940397 (+ 8983us) server_negotiation.cc:407] Connection header received
0928 02:05:09.941146 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:09.941155 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:09.941268 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:09.941457 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:09.944533 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.945684 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.948881 (+ 3197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.949510 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.952937 (+ 3427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:09.955447 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:09.955456 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:09.955465 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:09.955541 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:09.963708 (+ 8167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:09.964974 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:09.964980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:09.964983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:09.965128 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:09.965627 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:09.965635 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:09.965641 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:09.966072 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:09.968948 (+ 2876us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:09.969105 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 02:05:09.969382 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12368,"thread_start_us":12577,"threads_started":1}
I20260928 02:05:09.970342 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:09.918590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45051 (local address 127.1.182.131:53743)
0928 02:05:09.935840 (+ 17250us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:09.935900 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:09.940609 (+ 4709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:09.943432 (+ 2823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:09.943436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:09.943855 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:09.944355 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:09.944364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.947131 (+ 2767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.947136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:09.948786 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:09.948804 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.949682 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.949692 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:09.950015 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:09.951000 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:09.951033 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:09.956285 (+ 5252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:09.961087 (+ 4802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:09.961100 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:09.961125 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:09.961859 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:09.962379 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:09.962387 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:09.962393 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:09.962681 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:09.963638 (+ 957us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:09.963650 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:09.964606 (+ 956us) client_negotiation.cc:769] Sending connection context
0928 02:05:09.964997 (+ 391us) client_negotiation.cc:240] Negotiation successful
0928 02:05:09.965730 (+ 733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16913,"mutex_wait_us":23,"spinlock_wait_cycles":3840,"thread_start_us":21281,"threads_started":1}
I20260928 02:05:09.972287 9164 heartbeater.cc:347] Connected to a master server at 127.1.182.188:45051
I20260928 02:05:09.972637 9164 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:09.973621 9164 heartbeater.cc:511] Master 127.1.182.188:45051 requested a full tablet report, sending...
I20260928 02:05:09.974757 9174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:09.920886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:49645 (local address 127.1.182.190:39651)
0928 02:05:09.942010 (+ 21124us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:09.942015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:09.942030 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:05:09.942076 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:09.942079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:09.942132 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:09.942255 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:09.944525 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.945664 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.949820 (+ 4156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.950517 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.950666 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:09.961102 (+ 10436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:09.961111 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:09.961115 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:09.961158 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:09.965298 (+ 4140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:09.972423 (+ 7125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:09.972429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:09.972433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:09.972612 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:09.973109 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:09.973114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:09.973117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:09.973470 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:09.973636 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:09.974136 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 02:05:09.974382 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21015,"thread_start_us":20920,"threads_started":1}
I20260928 02:05:09.975831 8697 ts_manager.cc:194] Registered new tserver with Master: 5db2590c7c594cd9b59e518991ff1445 (127.1.182.131:33133)
I20260928 02:05:09.976533 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:09.919236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.131:49645)
0928 02:05:09.939569 (+ 20333us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:09.939597 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:09.940609 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:09.942281 (+ 1672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:09.942290 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:09.943160 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:09.944289 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:09.944326 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.945864 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.945869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:09.949039 (+ 3170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:09.949049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.950729 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.950738 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:09.951656 (+ 918us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:09.952335 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:09.952364 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:09.960809 (+ 8445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:09.969334 (+ 8525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:09.969339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:09.969342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:09.972286 (+ 2944us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:09.972756 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:09.972764 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:09.972770 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:09.972979 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:09.973609 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:09.973618 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:09.973836 (+ 218us) client_negotiation.cc:769] Sending connection context
0928 02:05:09.974018 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 02:05:09.974415 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20183,"mutex_wait_us":165,"thread_start_us":20008,"threads_started":1}
I20260928 02:05:09.977886 9165 heartbeater.cc:347] Connected to a master server at 127.1.182.190:39651
I20260928 02:05:09.978037 9165 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:09.978570 9165 heartbeater.cc:511] Master 127.1.182.190:39651 requested a full tablet report, sending...
I20260928 02:05:09.979841 7902 ts_manager.cc:194] Re-registered known tserver with Master: 5db2590c7c594cd9b59e518991ff1445 (127.1.182.131:33133)
I20260928 02:05:09.981392 7902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:49645
I20260928 02:05:09.987528 9166 heartbeater.cc:347] Connected to a master server at 127.1.182.189:39971
I20260928 02:05:09.987665 9166 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:09.988097 9166 heartbeater.cc:511] Master 127.1.182.189:39971 requested a full tablet report, sending...
I20260928 02:05:09.990653 8502 ts_manager.cc:194] Registered new tserver with Master: 5db2590c7c594cd9b59e518991ff1445 (127.1.182.131:33133)
I20260928 02:05:10.003052 9174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:09.951211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55232 (local address 127.1.182.190:39651)
0928 02:05:09.975242 (+ 24031us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:09.975253 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:09.983873 (+ 8620us) server_negotiation.cc:407] Connection header received
0928 02:05:09.984105 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:09.984110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:09.984203 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:09.984296 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:09.988621 (+ 4325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.989776 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.991727 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.992117 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.992247 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:09.997213 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:09.997217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:09.997220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:09.997251 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:10.000200 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:10.000846 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:10.000852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:10.000857 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:10.001002 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:10.001465 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:10.001469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:10.001473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:10.001856 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:10.002106 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:10.002442 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 02:05:10.002647 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23911,"thread_start_us":39710,"threads_started":1}
I20260928 02:05:10.004542 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:09.971387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41316 (local address 127.1.182.188:45051)
0928 02:05:09.971585 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:09.971593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:09.985651 (+ 14058us) server_negotiation.cc:407] Connection header received
0928 02:05:09.985731 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:09.985738 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:09.985851 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:09.985956 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:09.991583 (+ 5627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.992659 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.994200 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.994720 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.994860 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:09.997506 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:09.997512 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:09.997519 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:09.997554 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:10.000903 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:10.001632 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:10.001637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:10.001640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:10.001812 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:10.002226 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:10.002232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:10.002236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:10.002603 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:10.002975 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:10.003989 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0928 02:05:10.004209 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:05:10.004904 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:09.951270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42036 (local address 127.1.182.189:39971)
0928 02:05:09.970208 (+ 18938us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:09.970216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:09.984767 (+ 14551us) server_negotiation.cc:407] Connection header received
0928 02:05:09.984965 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:09.984973 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:09.985069 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:09.985163 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:09.988909 (+ 3746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.989946 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.992941 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:09.993364 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:09.993516 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:09.998321 (+ 4805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:09.998328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:09.998332 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:09.998374 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:10.001624 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:10.002428 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:10.002433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:10.002435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:10.002626 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:10.003139 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:10.003146 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:10.003152 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:10.003490 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:10.003739 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:10.004341 (+ 602us) server_negotiation.cc:299] Negotiation successful
0928 02:05:10.004557 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18794,"thread_start_us":35501,"threads_started":1}
I20260928 02:05:10.007562 7902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:05:10.015106 7902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:05:10.053757 9193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:10.041459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:39733 (local address 127.0.0.1:57278)
0928 02:05:10.042068 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:10.042087 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:10.042201 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:10.043029 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:10.043034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:10.043058 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:10.043458 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:10.043473 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.045954 (+ 2481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.045963 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:10.049459 (+ 3496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:10.049482 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.052622 (+ 3140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.052632 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:10.052888 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:10.052953 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 02:05:10.053063 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:05:10.053260 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":459,"threads_started":1}
I20260928 02:05:10.055088 9194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:10.041669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.1.182.130:39733)
0928 02:05:10.042525 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:10.042534 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:10.042566 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:05:10.042644 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:10.042650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:10.042850 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:10.043055 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:10.043655 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.045722 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.049699 (+ 3977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.052026 (+ 2327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.052201 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:10.052488 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:10.054431 (+ 1943us) server_negotiation.cc:299] Negotiation successful
0928 02:05:10.054673 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":607,"threads_started":1}
I20260928 02:05:10.069378 8957 tablet_service.cc:1467] Processing CreateTablet for tablet cd4394b7c1b84d56a044b45681eec272 (DEFAULT_TABLE table=test-table [id=688024d3249c42ac8c767f6c5eba394d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:05:10.070909 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd4394b7c1b84d56a044b45681eec272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:10.071879 9193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:10.042229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:33133 (local address 127.0.0.1:53578)
0928 02:05:10.054087 (+ 11858us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:10.054111 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:10.054233 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:10.065757 (+ 11524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:10.065762 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:10.065787 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:10.065973 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:10.065983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.068004 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.068009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:10.069398 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:10.069409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.070840 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.070958 (+ 118us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:10.071272 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:10.071306 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:05:10.071387 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:05:10.071536 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11726,"mutex_wait_us":1,"thread_start_us":22850,"threads_started":1}
I20260928 02:05:10.072001 9196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:10.042835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53578 (local address 127.1.182.131:33133)
0928 02:05:10.065046 (+ 22211us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:10.065059 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:10.065103 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:05:10.065237 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:10.065250 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:10.065548 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:10.065783 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:10.066136 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.067858 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.069551 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.070681 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.070807 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:10.071444 (+ 637us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:10.071530 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 02:05:10.071681 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21974,"thread_start_us":21743,"threads_started":1}
I20260928 02:05:10.073635 9197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:10.043454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:46607 (local address 127.0.0.1:46106)
0928 02:05:10.064142 (+ 20688us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:10.064172 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:10.064311 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:10.066478 (+ 2167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:10.066481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:10.066500 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:10.066731 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:10.066738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.068293 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.068298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:10.070365 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:10.070378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.072833 (+ 2455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.072838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:10.073044 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:10.073085 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 02:05:10.073176 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 02:05:10.073304 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20512,"mutex_wait_us":60,"thread_start_us":20295,"threads_started":1}
I20260928 02:05:10.074337 9198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:10.043677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46106 (local address 127.1.182.129:46607)
0928 02:05:10.065619 (+ 21942us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:10.065627 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:10.065654 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:05:10.065741 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:10.065749 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:10.066103 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:10.066294 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:10.066900 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.068137 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.070522 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.072399 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.072532 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:10.072722 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:10.073905 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0928 02:05:10.074083 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21780,"thread_start_us":21670,"threads_started":1}
I20260928 02:05:10.076154 9099 tablet_service.cc:1467] Processing CreateTablet for tablet cd4394b7c1b84d56a044b45681eec272 (DEFAULT_TABLE table=test-table [id=688024d3249c42ac8c767f6c5eba394d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:05:10.077255 9099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd4394b7c1b84d56a044b45681eec272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:10.080092 8817 tablet_service.cc:1467] Processing CreateTablet for tablet cd4394b7c1b84d56a044b45681eec272 (DEFAULT_TABLE table=test-table [id=688024d3249c42ac8c767f6c5eba394d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:05:10.081398 8817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd4394b7c1b84d56a044b45681eec272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:10.084129 9199 tablet_bootstrap.cc:492] T cd4394b7c1b84d56a044b45681eec272 P 5e5d0dbc247b46bc8219f4db2a911ec7: Bootstrap starting.
I20260928 02:05:10.087018 9199 tablet_bootstrap.cc:654] T cd4394b7c1b84d56a044b45681eec272 P 5e5d0dbc247b46bc8219f4db2a911ec7: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:10.088014 9199 log.cc:824] T cd4394b7c1b84d56a044b45681eec272 P 5e5d0dbc247b46bc8219f4db2a911ec7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:05:10.092743 9201 tablet_bootstrap.cc:492] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445: Bootstrap starting.
I20260928 02:05:10.094144 9199 tablet_bootstrap.cc:492] T cd4394b7c1b84d56a044b45681eec272 P 5e5d0dbc247b46bc8219f4db2a911ec7: No bootstrap required, opened a new log
I20260928 02:05:10.094436 9199 ts_tablet_manager.cc:1397] T cd4394b7c1b84d56a044b45681eec272 P 5e5d0dbc247b46bc8219f4db2a911ec7: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260928 02:05:10.096377 9201 tablet_bootstrap.cc:654] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:10.096634 9202 tablet_bootstrap.cc:492] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55: Bootstrap starting.
I20260928 02:05:10.098071 9201 log.cc:824] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445: Log is configured to *not* fsync() on all Append() calls
I20260928 02:05:10.101023 9202 tablet_bootstrap.cc:654] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:10.101284 9201 tablet_bootstrap.cc:492] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445: No bootstrap required, opened a new log
I20260928 02:05:10.101544 9201 ts_tablet_manager.cc:1397] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 02:05:10.102195 9202 log.cc:824] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55: Log is configured to *not* fsync() on all Append() calls
I20260928 02:05:10.104892 9202 tablet_bootstrap.cc:492] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55: No bootstrap required, opened a new log
I20260928 02:05:10.105139 9202 ts_tablet_manager.cc:1397] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260928 02:05:10.105566 9199 raft_consensus.cc:348] T cd4394b7c1b84d56a044b45681eec272 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39733 } } peers { permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 } } peers { permanent_uuid: "9d156a26f9e140dcb64782bed8729d55" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46607 } }
I20260928 02:05:10.106475 9199 raft_consensus.cc:374] T cd4394b7c1b84d56a044b45681eec272 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:10.106582 9199 raft_consensus.cc:729] T cd4394b7c1b84d56a044b45681eec272 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5d0dbc247b46bc8219f4db2a911ec7, State: Initialized, Role: FOLLOWER
I20260928 02:05:10.107293 9199 consensus_queue.cc:260] T cd4394b7c1b84d56a044b45681eec272 P 5e5d0dbc247b46bc8219f4db2a911ec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39733 } } peers { permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 } } peers { permanent_uuid: "9d156a26f9e140dcb64782bed8729d55" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46607 } }
I20260928 02:05:10.109004 9199 ts_tablet_manager.cc:1428] T cd4394b7c1b84d56a044b45681eec272 P 5e5d0dbc247b46bc8219f4db2a911ec7: Time spent starting tablet: real 0.014s user 0.005s sys 0.011s
I20260928 02:05:10.109321 9023 heartbeater.cc:503] Master 127.1.182.190:39651 was elected leader, sending a full tablet report...
I20260928 02:05:10.113255 9201 raft_consensus.cc:348] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39733 } } peers { permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 } } peers { permanent_uuid: "9d156a26f9e140dcb64782bed8729d55" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46607 } }
I20260928 02:05:10.113747 9201 raft_consensus.cc:374] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:10.113931 9201 raft_consensus.cc:729] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db2590c7c594cd9b59e518991ff1445, State: Initialized, Role: FOLLOWER
I20260928 02:05:10.114578 9201 consensus_queue.cc:260] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39733 } } peers { permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 } } peers { permanent_uuid: "9d156a26f9e140dcb64782bed8729d55" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46607 } }
I20260928 02:05:10.116333 9201 ts_tablet_manager.cc:1428] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260928 02:05:10.116755 9165 heartbeater.cc:503] Master 127.1.182.190:39651 was elected leader, sending a full tablet report...
I20260928 02:05:10.118873 9202 raft_consensus.cc:348] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39733 } } peers { permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 } } peers { permanent_uuid: "9d156a26f9e140dcb64782bed8729d55" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46607 } }
I20260928 02:05:10.119402 9202 raft_consensus.cc:374] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:10.119539 9202 raft_consensus.cc:729] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d156a26f9e140dcb64782bed8729d55, State: Initialized, Role: FOLLOWER
I20260928 02:05:10.120071 9202 consensus_queue.cc:260] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39733 } } peers { permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 } } peers { permanent_uuid: "9d156a26f9e140dcb64782bed8729d55" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46607 } }
I20260928 02:05:10.121605 9202 ts_tablet_manager.cc:1428] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55: Time spent starting tablet: real 0.016s user 0.009s sys 0.004s
I20260928 02:05:10.157953 9207 raft_consensus.cc:482] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:10.158226 9207 raft_consensus.cc:504] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39733 } } peers { permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 } } peers { permanent_uuid: "9d156a26f9e140dcb64782bed8729d55" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46607 } }
I20260928 02:05:10.159804 9207 leader_election.cc:288] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e5d0dbc247b46bc8219f4db2a911ec7 (127.1.182.130:39733), 5db2590c7c594cd9b59e518991ff1445 (127.1.182.131:33133)
I20260928 02:05:10.166890 9194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:10.160229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:36923 (local address 127.1.182.130:39733)
0928 02:05:10.160452 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:10.160459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:10.160725 (+ 266us) server_negotiation.cc:407] Connection header received
0928 02:05:10.161070 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:10.161079 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:10.161167 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:10.161262 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:10.161925 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.162887 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.165203 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.165961 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.166072 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:10.166179 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:10.166496 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 02:05:10.166622 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:05:10.166893 9196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:10.160982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:35503 (local address 127.1.182.131:33133)
0928 02:05:10.161196 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:10.161201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:10.161507 (+ 306us) server_negotiation.cc:407] Connection header received
0928 02:05:10.161648 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:10.161654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:10.161734 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:10.161829 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:10.162336 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.163136 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.165257 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.166001 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.166180 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:10.166251 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:10.166535 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 02:05:10.166647 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:05:10.167057 9209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:10.160834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:33133 (local address 127.1.182.129:35503)
0928 02:05:10.161387 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:10.161406 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:10.161531 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:10.161867 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:10.161872 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:10.161899 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:10.162198 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:10.162208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.163273 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.163277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:10.165078 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:10.165089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.166238 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.166244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:10.166361 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:10.166382 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:05:10.166495 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 02:05:10.166613 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":294,"threads_started":1}
I20260928 02:05:10.167124 9208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:10.160053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:39733 (local address 127.1.182.129:36923)
0928 02:05:10.160617 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:10.160633 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:10.160793 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:10.161348 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:10.161352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:10.161379 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:10.161779 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:10.161794 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.163028 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.163033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:10.165062 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:10.165073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.166160 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.166167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:10.166340 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:10.166363 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:05:10.166436 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:05:10.166576 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":27,"thread_start_us":326,"threads_started":1}
I20260928 02:05:10.168120 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4394b7c1b84d56a044b45681eec272" candidate_uuid: "9d156a26f9e140dcb64782bed8729d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db2590c7c594cd9b59e518991ff1445" is_pre_election: true
I20260928 02:05:10.168354 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4394b7c1b84d56a044b45681eec272" candidate_uuid: "9d156a26f9e140dcb64782bed8729d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" is_pre_election: true
I20260928 02:05:10.168697 9119 raft_consensus.cc:2454] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d156a26f9e140dcb64782bed8729d55 in term 0.
I20260928 02:05:10.168826 8977 raft_consensus.cc:2454] T cd4394b7c1b84d56a044b45681eec272 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d156a26f9e140dcb64782bed8729d55 in term 0.
I20260928 02:05:10.169386 8763 leader_election.cc:302] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5db2590c7c594cd9b59e518991ff1445, 9d156a26f9e140dcb64782bed8729d55; no voters:
I20260928 02:05:10.169714 9207 raft_consensus.cc:2781] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:05:10.169860 9207 raft_consensus.cc:482] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:05:10.169952 9207 raft_consensus.cc:3037] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:10.172122 9207 raft_consensus.cc:504] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39733 } } peers { permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 } } peers { permanent_uuid: "9d156a26f9e140dcb64782bed8729d55" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46607 } }
I20260928 02:05:10.172678 9207 leader_election.cc:288] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 [CANDIDATE]: Term 1 election: Requested vote from peers 5e5d0dbc247b46bc8219f4db2a911ec7 (127.1.182.130:39733), 5db2590c7c594cd9b59e518991ff1445 (127.1.182.131:33133)
I20260928 02:05:10.173067 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4394b7c1b84d56a044b45681eec272" candidate_uuid: "9d156a26f9e140dcb64782bed8729d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7"
I20260928 02:05:10.173180 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4394b7c1b84d56a044b45681eec272" candidate_uuid: "9d156a26f9e140dcb64782bed8729d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db2590c7c594cd9b59e518991ff1445"
I20260928 02:05:10.173269 8977 raft_consensus.cc:3037] T cd4394b7c1b84d56a044b45681eec272 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:10.173403 9119 raft_consensus.cc:3037] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:10.175498 8977 raft_consensus.cc:2454] T cd4394b7c1b84d56a044b45681eec272 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d156a26f9e140dcb64782bed8729d55 in term 1.
I20260928 02:05:10.175657 9119 raft_consensus.cc:2454] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d156a26f9e140dcb64782bed8729d55 in term 1.
I20260928 02:05:10.175872 8763 leader_election.cc:302] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5d0dbc247b46bc8219f4db2a911ec7, 9d156a26f9e140dcb64782bed8729d55; no voters:
I20260928 02:05:10.176187 9207 raft_consensus.cc:2781] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:05:10.176600 9207 raft_consensus.cc:686] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 [term 1 LEADER]: Becoming Leader. State: Replica: 9d156a26f9e140dcb64782bed8729d55, State: Running, Role: LEADER
I20260928 02:05:10.177146 9207 consensus_queue.cc:237] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39733 } } peers { permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 } } peers { permanent_uuid: "9d156a26f9e140dcb64782bed8729d55" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46607 } }
I20260928 02:05:10.182544 7902 catalog_manager.cc:5495] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d156a26f9e140dcb64782bed8729d55 (127.1.182.129). New cstate: current_term: 1 leader_uuid: "9d156a26f9e140dcb64782bed8729d55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d156a26f9e140dcb64782bed8729d55" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46607 } health_report { overall_health: HEALTHY } } }
W20260928 02:05:10.192005 9167 tablet.cc:2369] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:05:10.206972 8885 tablet.cc:2369] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:05:10.235864 9198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:10.230670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46114 (local address 127.1.182.129:46607)
0928 02:05:10.230878 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:10.230885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:10.230904 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:05:10.231005 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:10.231009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:10.231160 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:10.231313 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:05:10.231685 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.232385 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.233690 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.234100 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.234235 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:10.234705 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:05:10.235293 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:05:10.235477 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:10.235536 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 02:05:10.235614 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:05:10.244179 9119 raft_consensus.cc:1261] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445 [term 1 FOLLOWER]: Refusing update from remote peer 9d156a26f9e140dcb64782bed8729d55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:05:10.244239 8977 raft_consensus.cc:1261] T cd4394b7c1b84d56a044b45681eec272 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 1 FOLLOWER]: Refusing update from remote peer 9d156a26f9e140dcb64782bed8729d55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:05:10.245198 9210 consensus_queue.cc:1035] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:05:10.245594 9207 consensus_queue.cc:1035] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:05:10.282536 7903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:05:10.282789 7903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:05:10.290017 7903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bed972ae228a4bca907f46323a578168: Sending DeleteTablet for 3 replicas of tablet cd4394b7c1b84d56a044b45681eec272
I20260928 02:05:10.291471 8817 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4394b7c1b84d56a044b45681eec272 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:05:10 UTC) from {username='oryx', principal='oryx/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:46106
I20260928 02:05:10.291813 8957 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4394b7c1b84d56a044b45681eec272 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:05:10 UTC) from {username='oryx', principal='oryx/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:57278
I20260928 02:05:10.292584 9099 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4394b7c1b84d56a044b45681eec272 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:05:10 UTC) from {username='oryx', principal='oryx/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:53578
I20260928 02:05:10.302726 9224 tablet_replica.cc:330] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445: stopping tablet replica
I20260928 02:05:10.303207 9222 tablet_replica.cc:330] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55: stopping tablet replica
I20260928 02:05:10.303692 9223 tablet_replica.cc:330] T cd4394b7c1b84d56a044b45681eec272 P 5e5d0dbc247b46bc8219f4db2a911ec7: stopping tablet replica
I20260928 02:05:10.303694 9224 raft_consensus.cc:2229] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:05:10.304164 9223 raft_consensus.cc:2229] T cd4394b7c1b84d56a044b45681eec272 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:05:10.304611 9224 raft_consensus.cc:2258] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:05:10.304638 9223 raft_consensus.cc:2258] T cd4394b7c1b84d56a044b45681eec272 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:05:10.305269 9222 raft_consensus.cc:2229] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:05:10.306011 9222 raft_consensus.cc:2258] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:05:10.307929 9224 ts_tablet_manager.cc:1905] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:10.308097 9223 ts_tablet_manager.cc:1905] T cd4394b7c1b84d56a044b45681eec272 P 5e5d0dbc247b46bc8219f4db2a911ec7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:10.309056 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 7871
I20260928 02:05:10.310945 9222 ts_tablet_manager.cc:1905] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:10.311738 7933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:10.315749 9223 ts_tablet_manager.cc:1918] T cd4394b7c1b84d56a044b45681eec272 P 5e5d0dbc247b46bc8219f4db2a911ec7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:05:10.315944 9223 log.cc:1196] T cd4394b7c1b84d56a044b45681eec272 P 5e5d0dbc247b46bc8219f4db2a911ec7: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/cd4394b7c1b84d56a044b45681eec272
I20260928 02:05:10.315968 9224 ts_tablet_manager.cc:1918] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:05:10.316148 9224 log.cc:1196] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/cd4394b7c1b84d56a044b45681eec272
I20260928 02:05:10.316548 9223 ts_tablet_manager.cc:1939] T cd4394b7c1b84d56a044b45681eec272 P 5e5d0dbc247b46bc8219f4db2a911ec7: Deleting consensus metadata
I20260928 02:05:10.316648 9224 ts_tablet_manager.cc:1939] T cd4394b7c1b84d56a044b45681eec272 P 5db2590c7c594cd9b59e518991ff1445: Deleting consensus metadata
I20260928 02:05:10.328646 9222 ts_tablet_manager.cc:1918] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:05:10.328868 9222 log.cc:1196] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/wal/wals/cd4394b7c1b84d56a044b45681eec272
I20260928 02:05:10.329473 9222 ts_tablet_manager.cc:1939] T cd4394b7c1b84d56a044b45681eec272 P 9d156a26f9e140dcb64782bed8729d55: Deleting consensus metadata
I20260928 02:05:10.346583 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:10.294975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42048 (local address 127.1.182.189:39971)
0928 02:05:10.295206 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:10.295215 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:10.307287 (+ 12072us) server_negotiation.cc:407] Connection header received
0928 02:05:10.307796 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:10.307802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:10.307884 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:10.308003 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:10.324882 (+ 16879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.326460 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.330742 (+ 4282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.331670 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.331825 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:10.336991 (+ 5166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:10.337001 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:10.337009 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:10.337062 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:10.341707 (+ 4645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:10.342811 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:10.342820 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:10.342826 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:10.343028 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:10.344658 (+ 1630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:10.344666 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:10.344689 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:10.345197 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:10.345424 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:10.345995 (+ 571us) server_negotiation.cc:299] Negotiation successful
0928 02:05:10.346225 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:05:10.365720 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:10.307530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.1.182.188:45051)
0928 02:05:10.307706 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:10.307711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:10.349204 (+ 41493us) server_negotiation.cc:407] Connection header received
0928 02:05:10.349451 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:10.349458 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:10.349539 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:10.349609 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:10.351583 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.352712 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.354425 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.355124 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.355253 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:10.358841 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:10.358847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:10.358852 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:10.358892 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:10.362761 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:10.363660 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:10.363667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:10.363673 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:10.363824 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:10.364224 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:10.364228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:10.364231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:10.364563 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:10.364811 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:10.365133 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 02:05:10.365345 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:05:10.467743 7887 catalog_manager.cc:4841] TS 9d156a26f9e140dcb64782bed8729d55 (127.1.182.129:46607): tablet cd4394b7c1b84d56a044b45681eec272 (table test-table [id=688024d3249c42ac8c767f6c5eba394d]) successfully deleted
W20260928 02:05:10.468062 7887 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.182.129:46607, user_credentials={real_user=slave}} blocked reactor thread for 136832us
I20260928 02:05:10.468212 7888 catalog_manager.cc:4841] TS 5db2590c7c594cd9b59e518991ff1445 (127.1.182.131:33133): tablet cd4394b7c1b84d56a044b45681eec272 (table test-table [id=688024d3249c42ac8c767f6c5eba394d]) successfully deleted
W20260928 02:05:10.468466 7888 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.182.131:33133, user_credentials={real_user=slave}} blocked reactor thread for 150143us
I20260928 02:05:10.468750 7888 catalog_manager.cc:4841] TS 5e5d0dbc247b46bc8219f4db2a911ec7 (127.1.182.130:39733): tablet cd4394b7c1b84d56a044b45681eec272 (table test-table [id=688024d3249c42ac8c767f6c5eba394d]) successfully deleted
I20260928 02:05:10.469365 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 7871
I20260928 02:05:10.531162 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:10.509458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41336 (local address 127.1.182.188:45051)
0928 02:05:10.509676 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:10.509684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:10.514330 (+ 4646us) server_negotiation.cc:407] Connection header received
0928 02:05:10.514618 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:10.514626 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:10.514716 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:10.514799 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:10.517020 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.518104 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.521148 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.521802 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.521933 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:10.525017 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:10.525028 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:10.525032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:10.525069 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:10.527998 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:10.529005 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:10.529010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:10.529013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:10.529157 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:10.529637 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:10.529642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:10.529645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:10.529918 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:10.530169 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:10.530505 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 02:05:10.530729 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:05:10.531535 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:10.509690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42060 (local address 127.1.182.189:39971)
0928 02:05:10.509932 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:10.509940 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:10.514451 (+ 4511us) server_negotiation.cc:407] Connection header received
0928 02:05:10.514955 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:10.514961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:10.515039 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:10.515173 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:10.517244 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.517952 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.520103 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:10.520824 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:10.520966 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:10.524499 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:10.524506 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:10.524511 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:10.524551 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:10.528013 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:10.529010 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:10.529015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:10.529019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:10.529194 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:10.529613 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:10.529618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:10.529622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:10.529988 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:10.530238 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:10.530690 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 02:05:10.531169 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:05:11.270975 9234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.270216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.130:35295)
0928 02:05:11.270695 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.270814 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":255,"threads_started":1}
W20260928 02:05:11.271682 9023 heartbeater.cc:644] Failed to heartbeat to 127.1.182.190:39651 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
I20260928 02:05:11.282594 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.276568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.131:37785)
0928 02:05:11.282276 (+ 5708us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.282385 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5608,"threads_started":1}
W20260928 02:05:11.283303 9165 heartbeater.cc:644] Failed to heartbeat to 127.1.182.190:39651 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
I20260928 02:05:11.283874 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.271435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.129:42395)
0928 02:05:11.283513 (+ 12078us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.283661 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11947,"thread_start_us":11843,"threads_started":1}
W20260928 02:05:11.284363 8883 heartbeater.cc:644] Failed to heartbeat to 127.1.182.190:39651 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
I20260928 02:05:11.789639 9237 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bed972ae228a4bca907f46323a578168)
I20260928 02:05:11.789698 9238 raft_consensus.cc:482] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bed972ae228a4bca907f46323a578168)
I20260928 02:05:11.789857 9237 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: false } }
I20260928 02:05:11.789973 9238 raft_consensus.cc:504] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: false } }
I20260928 02:05:11.790753 9237 leader_election.cc:288] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bed972ae228a4bca907f46323a578168 (127.1.182.190:39651), d64d03577e9843debd2679fd7e2b7ec6 (127.1.182.189:39971)
I20260928 02:05:11.798146 9238 leader_election.cc:288] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bed972ae228a4bca907f46323a578168 (127.1.182.190:39651), 6fb0862e5a314154968a6294cee9cc6a (127.1.182.188:45051)
I20260928 02:05:11.799089 9239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.790899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.0.0.1:55638)
0928 02:05:11.791608 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.791845 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":422,"threads_started":1}
W20260928 02:05:11.799502 8681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
I20260928 02:05:11.803681 9239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.803281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.0.0.1:55666)
0928 02:05:11.803443 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.803536 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 02:05:11.803998 8681 leader_election.cc:334] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bed972ae228a4bca907f46323a578168 (127.1.182.190:39651): Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
I20260928 02:05:11.803409 9243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.791643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.0.0.1:55650)
0928 02:05:11.801912 (+ 10269us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.802749 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10117,"thread_start_us":11214,"threads_started":1}
W20260928 02:05:11.804610 8486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
I20260928 02:05:11.806565 9243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.805783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.0.0.1:55682)
0928 02:05:11.806145 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.806306 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260928 02:05:11.806834 8486 leader_election.cc:334] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bed972ae228a4bca907f46323a578168 (127.1.182.190:39651): Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
I20260928 02:05:11.808058 9241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.791302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42070 (local address 127.1.182.189:39971)
0928 02:05:11.792557 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:11.792566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:11.799675 (+ 7109us) server_negotiation.cc:407] Connection header received
0928 02:05:11.799761 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:11.799766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:11.799864 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:11.799970 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:11.801546 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.803060 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.805688 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.807209 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.807348 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:11.807414 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:11.807723 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 02:05:11.807811 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1078,"thread_start_us":881,"threads_started":1}
I20260928 02:05:11.808139 9240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.791008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39971 (local address 127.0.0.1:42070)
0928 02:05:11.798964 (+ 7956us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.798983 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:11.799162 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:11.800561 (+ 1399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:11.800569 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:11.800609 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:11.801056 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:11.801073 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.803279 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.803285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:11.805481 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:11.805494 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.807357 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.807363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:11.807552 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:11.807590 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:05:11.807683 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 02:05:11.807812 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7842,"thread_start_us":7690,"threads_started":1}
I20260928 02:05:11.808750 8512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fb0862e5a314154968a6294cee9cc6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d64d03577e9843debd2679fd7e2b7ec6" is_pre_election: true
I20260928 02:05:11.809056 8512 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fb0862e5a314154968a6294cee9cc6a in term 1.
I20260928 02:05:11.809546 8682 leader_election.cc:302] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6fb0862e5a314154968a6294cee9cc6a, d64d03577e9843debd2679fd7e2b7ec6; no voters: bed972ae228a4bca907f46323a578168
I20260928 02:05:11.809918 9237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 02:05:11.810058 9237 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 FOLLOWER]: Starting leader election (detected failure of leader bed972ae228a4bca907f46323a578168)
I20260928 02:05:11.810174 9237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:05:11.811219 9242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.791753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45051 (local address 127.0.0.1:41348)
0928 02:05:11.803633 (+ 11880us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.803648 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:11.803786 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:11.804177 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:11.804182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:11.804211 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:11.804699 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:11.804710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.806298 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.806304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:11.808718 (+ 2414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:11.808731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.810396 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.810403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:11.810560 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:11.810581 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:05:11.810682 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 02:05:11.810801 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11733,"mutex_wait_us":1,"thread_start_us":9871,"threads_started":1}
I20260928 02:05:11.811230 9244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.792800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41348 (local address 127.1.182.188:45051)
0928 02:05:11.802284 (+ 9484us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:11.802292 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:11.803729 (+ 1437us) server_negotiation.cc:407] Connection header received
0928 02:05:11.803931 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:11.803936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:11.804022 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:11.804135 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:11.804895 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.806062 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.808870 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.810187 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.810342 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:11.810455 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:11.810703 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 02:05:11.810833 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9309,"thread_start_us":9794,"threads_started":1}
I20260928 02:05:11.811977 8706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d64d03577e9843debd2679fd7e2b7ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6fb0862e5a314154968a6294cee9cc6a" is_pre_election: true
I20260928 02:05:11.812856 9237 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: false } }
I20260928 02:05:11.813311 8706 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d64d03577e9843debd2679fd7e2b7ec6 in current term 2: Already voted for candidate 6fb0862e5a314154968a6294cee9cc6a in this term.
I20260928 02:05:11.813503 9237 leader_election.cc:288] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [CANDIDATE]: Term 2 election: Requested vote from peers bed972ae228a4bca907f46323a578168 (127.1.182.190:39651), d64d03577e9843debd2679fd7e2b7ec6 (127.1.182.189:39971)
I20260928 02:05:11.813958 8512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fb0862e5a314154968a6294cee9cc6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d64d03577e9843debd2679fd7e2b7ec6"
I20260928 02:05:11.814153 9240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.813581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.0.0.1:55692)
0928 02:05:11.813942 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.814028 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251}
I20260928 02:05:11.814229 8512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:05:11.814240 8489 leader_election.cc:302] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d64d03577e9843debd2679fd7e2b7ec6; no voters: 6fb0862e5a314154968a6294cee9cc6a, bed972ae228a4bca907f46323a578168
I20260928 02:05:11.815553 9240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.815148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.0.0.1:55708)
0928 02:05:11.815314 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.815420 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260928 02:05:11.815728 8681 leader_election.cc:334] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bed972ae228a4bca907f46323a578168 (127.1.182.190:39651): Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
I20260928 02:05:11.816803 8512 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fb0862e5a314154968a6294cee9cc6a in term 2.
I20260928 02:05:11.816977 9238 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:05:11.817912 8682 leader_election.cc:302] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6fb0862e5a314154968a6294cee9cc6a, d64d03577e9843debd2679fd7e2b7ec6; no voters: bed972ae228a4bca907f46323a578168
I20260928 02:05:11.818208 9237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:05:11.818709 9237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [term 2 LEADER]: Becoming Leader. State: Replica: 6fb0862e5a314154968a6294cee9cc6a, State: Running, Role: LEADER
I20260928 02:05:11.819128 9237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: false } }
I20260928 02:05:11.820847 9247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fb0862e5a314154968a6294cee9cc6a. Latest consensus state: current_term: 2 leader_uuid: "6fb0862e5a314154968a6294cee9cc6a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: false } } }
I20260928 02:05:11.821238 9247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:11.821856 9249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:05:11.825359 9249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:05:11.826171 9249 catalog_manager.cc:1259] Loaded cluster ID: 341136e4888648f094f3e854341582e0
I20260928 02:05:11.826239 9249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:05:11.827422 9249 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:05:11.828178 9249 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: Loaded TSK: 0
I20260928 02:05:11.828979 9249 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:05:11.877857 8696 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:05:11.885003 8696 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:05:11.889132 9240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.888728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.0.0.1:55738)
0928 02:05:11.888893 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.888997 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 02:05:11.889263 8512 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [term 2 FOLLOWER]: Refusing update from remote peer 6fb0862e5a314154968a6294cee9cc6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 02:05:11.890040 9247 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 02:05:11.890268 9240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.889772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.0.0.1:55746)
0928 02:05:11.890005 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.890100 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 02:05:11.890429 8681 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a -> Peer bed972ae228a4bca907f46323a578168 (127.1.182.190:39651): Couldn't send request to peer bed972ae228a4bca907f46323a578168. Status: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:05:11.893060 9238 sys_catalog.cc:455] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fb0862e5a314154968a6294cee9cc6a. Latest consensus state: current_term: 2 leader_uuid: "6fb0862e5a314154968a6294cee9cc6a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: false } } }
I20260928 02:05:11.893350 9238 sys_catalog.cc:458] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:05:11.894763 9247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6fb0862e5a314154968a6294cee9cc6a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: false } } }
I20260928 02:05:11.895040 9247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:11.895424 9251 mvcc.cc:205] Tried to move back new op lower bound from 7334138314291634176 to 7334138314015023104. Current Snapshot: MvccSnapshot[applied={T|T < 7334138314291634176}]
I20260928 02:05:11.895481 9238 sys_catalog.cc:455] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6fb0862e5a314154968a6294cee9cc6a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bed972ae228a4bca907f46323a578168" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39651 } } peers { permanent_uuid: "d64d03577e9843debd2679fd7e2b7ec6" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39971 } attrs { promote: false } } peers { permanent_uuid: "6fb0862e5a314154968a6294cee9cc6a" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45051 } attrs { promote: false } } }
I20260928 02:05:11.895711 9238 sys_catalog.cc:458] T 00000000000000000000000000000000 P d64d03577e9843debd2679fd7e2b7ec6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:05:11.913921 9239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.903636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:33133 (local address 127.0.0.1:53592)
0928 02:05:11.903767 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.903783 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:11.903903 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:11.904641 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:11.904646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:11.904671 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:11.905057 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:11.905069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.907111 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.907119 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:11.910431 (+ 3312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:11.910454 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.912736 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.912747 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:11.913137 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:11.913199 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 02:05:11.913291 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 02:05:11.913518 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":6}
I20260928 02:05:11.915948 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.903754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53592 (local address 127.1.182.131:33133)
0928 02:05:11.904266 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:11.904272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:11.904292 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:05:11.904354 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:11.904358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:11.904492 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:11.905197 (+ 705us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:11.905488 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.906918 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.910641 (+ 3723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.912398 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.915046 (+ 2648us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:11.915237 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:11.915302 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 02:05:11.915530 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":675,"threads_started":1}
I20260928 02:05:11.917018 9099 tablet_service.cc:1467] Processing CreateTablet for tablet 096573ae193e40c7b205161e88f33fd8 (DEFAULT_TABLE table=test-table [id=88908ec7eae846358747688ec0b02136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:05:11.917630 9099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096573ae193e40c7b205161e88f33fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:11.921271 9240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.903337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:39733 (local address 127.0.0.1:57284)
0928 02:05:11.903502 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.903523 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:11.903624 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:11.904716 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:11.904721 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:11.904753 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:11.905009 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:11.905030 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.907004 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.907013 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:11.914325 (+ 7312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:11.914347 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.920448 (+ 6101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.920455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:11.920666 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:11.920690 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:05:11.920814 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 02:05:11.920957 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":1156,"spinlock_wait_cycles":10112}
I20260928 02:05:11.921308 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.903927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57284 (local address 127.1.182.130:39733)
0928 02:05:11.904393 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:11.904400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:11.904418 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:05:11.904489 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:11.904493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:11.904567 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:11.905172 (+ 605us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:11.905380 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.906786 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.919465 (+ 12679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.920262 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.920419 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:11.920521 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:11.920845 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 02:05:11.920995 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":227,"threads_started":1}
I20260928 02:05:11.922147 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 096573ae193e40c7b205161e88f33fd8 (DEFAULT_TABLE table=test-table [id=88908ec7eae846358747688ec0b02136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:05:11.922607 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096573ae193e40c7b205161e88f33fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:11.925140 9260 tablet_bootstrap.cc:492] T 096573ae193e40c7b205161e88f33fd8 P 5db2590c7c594cd9b59e518991ff1445: Bootstrap starting.
I20260928 02:05:11.928406 9239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.904957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:46607 (local address 127.0.0.1:46118)
0928 02:05:11.914685 (+ 9728us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.914707 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:11.914834 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:11.923198 (+ 8364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:11.923202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:11.923225 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:11.923357 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:11.923365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.924330 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.924339 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:11.926278 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:11.926293 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.927663 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.927668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:11.927854 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:11.927891 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:05:11.927989 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 02:05:11.928129 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9609,"thread_start_us":12651,"threads_started":1}
I20260928 02:05:11.928606 9260 tablet_bootstrap.cc:654] T 096573ae193e40c7b205161e88f33fd8 P 5db2590c7c594cd9b59e518991ff1445: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:11.928620 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.905082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46118 (local address 127.1.182.129:46607)
0928 02:05:11.920868 (+ 15786us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:11.920875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:11.920894 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:05:11.920972 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:11.920978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:11.921088 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:11.921188 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:11.923518 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.924189 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.926424 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.927514 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.927654 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:11.927761 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:11.928042 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 02:05:11.928201 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15650,"thread_start_us":16204,"threads_started":1}
I20260928 02:05:11.929503 8817 tablet_service.cc:1467] Processing CreateTablet for tablet 096573ae193e40c7b205161e88f33fd8 (DEFAULT_TABLE table=test-table [id=88908ec7eae846358747688ec0b02136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:05:11.930044 8817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096573ae193e40c7b205161e88f33fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:11.930600 9260 tablet_bootstrap.cc:492] T 096573ae193e40c7b205161e88f33fd8 P 5db2590c7c594cd9b59e518991ff1445: No bootstrap required, opened a new log
I20260928 02:05:11.930755 9260 ts_tablet_manager.cc:1397] T 096573ae193e40c7b205161e88f33fd8 P 5db2590c7c594cd9b59e518991ff1445: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260928 02:05:11.931560 9261 tablet_bootstrap.cc:492] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7: Bootstrap starting.
I20260928 02:05:11.931631 9260 raft_consensus.cc:348] T 096573ae193e40c7b205161e88f33fd8 P 5db2590c7c594cd9b59e518991ff1445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39733 } } peers { permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 } } peers { permanent_uuid: "9d156a26f9e140dcb64782bed8729d55" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46607 } }
I20260928 02:05:11.931869 9260 raft_consensus.cc:374] T 096573ae193e40c7b205161e88f33fd8 P 5db2590c7c594cd9b59e518991ff1445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:11.931994 9260 raft_consensus.cc:729] T 096573ae193e40c7b205161e88f33fd8 P 5db2590c7c594cd9b59e518991ff1445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db2590c7c594cd9b59e518991ff1445, State: Initialized, Role: FOLLOWER
I20260928 02:05:11.932255 9260 consensus_queue.cc:260] T 096573ae193e40c7b205161e88f33fd8 P 5db2590c7c594cd9b59e518991ff1445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39733 } } peers { permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 } } peers { permanent_uuid: "9d156a26f9e140dcb64782bed8729d55" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46607 } }
I20260928 02:05:11.933130 9260 ts_tablet_manager.cc:1428] T 096573ae193e40c7b205161e88f33fd8 P 5db2590c7c594cd9b59e518991ff1445: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:05:11.933440 9261 tablet_bootstrap.cc:654] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:11.936314 9164 heartbeater.cc:503] Master 127.1.182.188:45051 was elected leader, sending a full tablet report...
I20260928 02:05:11.942262 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.933792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.131:60301)
0928 02:05:11.941708 (+ 7916us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.941888 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7346,"spinlock_wait_cycles":16768,"thread_start_us":7329,"threads_started":1}
I20260928 02:05:11.943151 9261 tablet_bootstrap.cc:492] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7: No bootstrap required, opened a new log
I20260928 02:05:11.943368 9261 ts_tablet_manager.cc:1397] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.008s
I20260928 02:05:11.943737 9266 tablet_bootstrap.cc:492] T 096573ae193e40c7b205161e88f33fd8 P 9d156a26f9e140dcb64782bed8729d55: Bootstrap starting.
I20260928 02:05:11.944363 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.943273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.131:59149)
0928 02:05:11.944143 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.944234 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":777}
I20260928 02:05:11.944222 9261 raft_consensus.cc:348] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39733 } } peers { permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 } } peers { permanent_uuid: "9d156a26f9e140dcb64782bed8729d55" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46607 } }
I20260928 02:05:11.944482 9261 raft_consensus.cc:374] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:11.944592 9261 raft_consensus.cc:729] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5d0dbc247b46bc8219f4db2a911ec7, State: Initialized, Role: FOLLOWER
I20260928 02:05:11.944828 9261 consensus_queue.cc:260] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39733 } } peers { permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 } } peers { permanent_uuid: "9d156a26f9e140dcb64782bed8729d55" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46607 } }
I20260928 02:05:11.945657 9261 ts_tablet_manager.cc:1428] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:05:11.946272 9266 tablet_bootstrap.cc:654] T 096573ae193e40c7b205161e88f33fd8 P 9d156a26f9e140dcb64782bed8729d55: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:11.946837 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.945814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.130:42873)
0928 02:05:11.946400 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.946586 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":322,"threads_started":1}
I20260928 02:05:11.948021 9022 heartbeater.cc:503] Master 127.1.182.188:45051 was elected leader, sending a full tablet report...
I20260928 02:05:11.948288 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.947889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.130:58711)
0928 02:05:11.948092 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.948167 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260928 02:05:11.952276 9266 tablet_bootstrap.cc:492] T 096573ae193e40c7b205161e88f33fd8 P 9d156a26f9e140dcb64782bed8729d55: No bootstrap required, opened a new log
I20260928 02:05:11.952399 9266 ts_tablet_manager.cc:1397] T 096573ae193e40c7b205161e88f33fd8 P 9d156a26f9e140dcb64782bed8729d55: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260928 02:05:11.952996 9266 raft_consensus.cc:348] T 096573ae193e40c7b205161e88f33fd8 P 9d156a26f9e140dcb64782bed8729d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39733 } } peers { permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 } } peers { permanent_uuid: "9d156a26f9e140dcb64782bed8729d55" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46607 } }
I20260928 02:05:11.953161 9266 raft_consensus.cc:374] T 096573ae193e40c7b205161e88f33fd8 P 9d156a26f9e140dcb64782bed8729d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:11.953224 9266 raft_consensus.cc:729] T 096573ae193e40c7b205161e88f33fd8 P 9d156a26f9e140dcb64782bed8729d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d156a26f9e140dcb64782bed8729d55, State: Initialized, Role: FOLLOWER
I20260928 02:05:11.953451 9266 consensus_queue.cc:260] T 096573ae193e40c7b205161e88f33fd8 P 9d156a26f9e140dcb64782bed8729d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39733 } } peers { permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 } } peers { permanent_uuid: "9d156a26f9e140dcb64782bed8729d55" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46607 } }
I20260928 02:05:11.954648 9266 ts_tablet_manager.cc:1428] T 096573ae193e40c7b205161e88f33fd8 P 9d156a26f9e140dcb64782bed8729d55: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 02:05:11.955304 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.954338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.129:42133)
0928 02:05:11.954900 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.955063 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"threads_started":1}
I20260928 02:05:11.956283 8882 heartbeater.cc:503] Master 127.1.182.188:45051 was elected leader, sending a full tablet report...
I20260928 02:05:11.956897 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.956222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.129:42261)
0928 02:05:11.956672 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.956767 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374}
I20260928 02:05:11.981496 9267 raft_consensus.cc:482] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:11.981753 9267 raft_consensus.cc:504] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39733 } } peers { permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 } } peers { permanent_uuid: "9d156a26f9e140dcb64782bed8729d55" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46607 } }
I20260928 02:05:11.983420 9267 leader_election.cc:288] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5db2590c7c594cd9b59e518991ff1445 (127.1.182.131:33133), 9d156a26f9e140dcb64782bed8729d55 (127.1.182.129:46607)
I20260928 02:05:11.989732 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.983649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:46607 (local address 127.1.182.130:60333)
0928 02:05:11.983803 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.983824 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:11.983973 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:11.984509 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:11.984514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:11.984545 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:11.984856 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:11.984868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.986096 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.986102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:11.987697 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:11.987712 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.988955 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.988961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:11.989145 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:11.989184 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 02:05:11.989304 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 02:05:11.989474 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 02:05:11.989756 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.983923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:60333 (local address 127.1.182.129:46607)
0928 02:05:11.984126 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:11.984134 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:11.984155 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:05:11.984250 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:11.984257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:11.984365 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:11.984463 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:11.985017 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.985953 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.987844 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.988780 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.988900 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:11.988953 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:11.989331 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 02:05:11.989474 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 02:05:11.990338 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.983844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:58857 (local address 127.1.182.131:33133)
0928 02:05:11.984073 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:11.984081 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:11.984617 (+ 536us) server_negotiation.cc:407] Connection header received
0928 02:05:11.984871 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:11.984876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:11.984971 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:11.985057 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:11.985613 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.986484 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.988329 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.989325 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.989462 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:11.989528 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:11.989952 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 02:05:11.990090 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 02:05:11.990706 8837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096573ae193e40c7b205161e88f33fd8" candidate_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d156a26f9e140dcb64782bed8729d55" is_pre_election: true
I20260928 02:05:11.990907 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:11.983649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:33133 (local address 127.1.182.130:58857)
0928 02:05:11.984517 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:11.984535 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:11.984711 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:11.985099 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:11.985105 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:11.985132 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:11.985447 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:11.985457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.986615 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.986619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:11.988180 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:11.988194 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:11.989542 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:11.989547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:11.989754 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:11.989797 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:05:11.990541 (+ 744us) client_negotiation.cc:240] Negotiation successful
0928 02:05:11.990648 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":70,"spinlock_wait_cycles":2304,"thread_start_us":487,"threads_started":1}
I20260928 02:05:11.991127 8837 raft_consensus.cc:2454] T 096573ae193e40c7b205161e88f33fd8 P 9d156a26f9e140dcb64782bed8729d55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5d0dbc247b46bc8219f4db2a911ec7 in term 0.
I20260928 02:05:11.991547 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096573ae193e40c7b205161e88f33fd8" candidate_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db2590c7c594cd9b59e518991ff1445" is_pre_election: true
I20260928 02:05:11.991796 9119 raft_consensus.cc:2454] T 096573ae193e40c7b205161e88f33fd8 P 5db2590c7c594cd9b59e518991ff1445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5d0dbc247b46bc8219f4db2a911ec7 in term 0.
I20260928 02:05:11.991760 8910 leader_election.cc:302] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5d0dbc247b46bc8219f4db2a911ec7, 9d156a26f9e140dcb64782bed8729d55; no voters:
I20260928 02:05:11.992218 9267 raft_consensus.cc:2781] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:05:11.992342 9267 raft_consensus.cc:482] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:05:11.992491 9267 raft_consensus.cc:3037] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:11.994606 9267 raft_consensus.cc:504] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39733 } } peers { permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 } } peers { permanent_uuid: "9d156a26f9e140dcb64782bed8729d55" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46607 } }
I20260928 02:05:11.995297 9267 leader_election.cc:288] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 [CANDIDATE]: Term 1 election: Requested vote from peers 5db2590c7c594cd9b59e518991ff1445 (127.1.182.131:33133), 9d156a26f9e140dcb64782bed8729d55 (127.1.182.129:46607)
I20260928 02:05:11.995710 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096573ae193e40c7b205161e88f33fd8" candidate_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db2590c7c594cd9b59e518991ff1445"
I20260928 02:05:11.995738 8837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096573ae193e40c7b205161e88f33fd8" candidate_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d156a26f9e140dcb64782bed8729d55"
I20260928 02:05:11.995908 9119 raft_consensus.cc:3037] T 096573ae193e40c7b205161e88f33fd8 P 5db2590c7c594cd9b59e518991ff1445 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:11.995947 8837 raft_consensus.cc:3037] T 096573ae193e40c7b205161e88f33fd8 P 9d156a26f9e140dcb64782bed8729d55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:11.998494 9119 raft_consensus.cc:2454] T 096573ae193e40c7b205161e88f33fd8 P 5db2590c7c594cd9b59e518991ff1445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5d0dbc247b46bc8219f4db2a911ec7 in term 1.
I20260928 02:05:11.998514 8837 raft_consensus.cc:2454] T 096573ae193e40c7b205161e88f33fd8 P 9d156a26f9e140dcb64782bed8729d55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5d0dbc247b46bc8219f4db2a911ec7 in term 1.
I20260928 02:05:11.998894 8911 leader_election.cc:302] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5db2590c7c594cd9b59e518991ff1445, 5e5d0dbc247b46bc8219f4db2a911ec7; no voters:
I20260928 02:05:11.999231 9267 raft_consensus.cc:2781] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:05:11.999728 9267 raft_consensus.cc:686] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 1 LEADER]: Becoming Leader. State: Replica: 5e5d0dbc247b46bc8219f4db2a911ec7, State: Running, Role: LEADER
I20260928 02:05:12.000159 9267 consensus_queue.cc:237] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39733 } } peers { permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 } } peers { permanent_uuid: "9d156a26f9e140dcb64782bed8729d55" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46607 } }
I20260928 02:05:12.000686 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:12.000301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.130:39333)
0928 02:05:12.000449 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:12.000568 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 02:05:12.004403 8696 catalog_manager.cc:5495] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e5d0dbc247b46bc8219f4db2a911ec7 (127.1.182.130). New cstate: current_term: 1 leader_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d0dbc247b46bc8219f4db2a911ec7" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d156a26f9e140dcb64782bed8729d55" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46607 } health_report { overall_health: UNKNOWN } } }
I20260928 02:05:12.028633 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:12.023696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57296 (local address 127.1.182.130:39733)
0928 02:05:12.023875 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:12.023880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:12.023896 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:05:12.024006 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:12.024010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:12.024082 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:12.024206 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:05:12.024556 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:12.025233 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:12.026481 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:12.026900 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:12.026999 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:12.027452 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:05:12.027991 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:05:12.028108 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:12.028270 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 02:05:12.028336 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:05:12.033171 9119 raft_consensus.cc:1261] T 096573ae193e40c7b205161e88f33fd8 P 5db2590c7c594cd9b59e518991ff1445 [term 1 FOLLOWER]: Refusing update from remote peer 5e5d0dbc247b46bc8219f4db2a911ec7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:05:12.033617 8837 raft_consensus.cc:1261] T 096573ae193e40c7b205161e88f33fd8 P 9d156a26f9e140dcb64782bed8729d55 [term 1 FOLLOWER]: Refusing update from remote peer 5e5d0dbc247b46bc8219f4db2a911ec7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:05:12.033967 9273 consensus_queue.cc:1035] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db2590c7c594cd9b59e518991ff1445" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:05:12.034411 9267 consensus_queue.cc:1035] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d156a26f9e140dcb64782bed8729d55" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:05:12.038393 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:12.037189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.131:54745)
0928 02:05:12.038026 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:12.038147 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260928 02:05:12.038856 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:12.037433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.129:56201)
0928 02:05:12.038571 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:12.038671 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331}
I20260928 02:05:12.048177 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:12.047776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.129:46745)
0928 02:05:12.047940 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:12.048024 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 02:05:12.054775 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:12.054278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.131:57197)
0928 02:05:12.054506 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:12.054601 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260928 02:05:12.057826 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:12.056854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.130:34237)
0928 02:05:12.057511 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:12.057621 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506}
I20260928 02:05:12.063040 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:12.062582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.1.182.130:49243)
0928 02:05:12.062762 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:12.062856 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 02:05:12.067754 8695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:05:12.068033 8695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:05:12.073992 8695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fb0862e5a314154968a6294cee9cc6a: Sending DeleteTablet for 3 replicas of tablet 096573ae193e40c7b205161e88f33fd8
I20260928 02:05:12.075174 9099 tablet_service.cc:1514] Processing DeleteTablet for tablet 096573ae193e40c7b205161e88f33fd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:05:12 UTC) from {username='oryx', principal='oryx/127.1.182.188@KRBTEST.COM'} at 127.0.0.1:53592
I20260928 02:05:12.075265 8957 tablet_service.cc:1514] Processing DeleteTablet for tablet 096573ae193e40c7b205161e88f33fd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:05:12 UTC) from {username='oryx', principal='oryx/127.1.182.188@KRBTEST.COM'} at 127.0.0.1:57284
I20260928 02:05:12.075768 8817 tablet_service.cc:1514] Processing DeleteTablet for tablet 096573ae193e40c7b205161e88f33fd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:05:12 UTC) from {username='oryx', principal='oryx/127.1.182.188@KRBTEST.COM'} at 127.0.0.1:46118
I20260928 02:05:12.078331 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 8671
I20260928 02:05:12.084646 9287 tablet_replica.cc:330] T 096573ae193e40c7b205161e88f33fd8 P 9d156a26f9e140dcb64782bed8729d55: stopping tablet replica
I20260928 02:05:12.084860 9287 raft_consensus.cc:2229] T 096573ae193e40c7b205161e88f33fd8 P 9d156a26f9e140dcb64782bed8729d55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:05:12.085122 9286 tablet_replica.cc:330] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7: stopping tablet replica
I20260928 02:05:12.085160 9287 raft_consensus.cc:2258] T 096573ae193e40c7b205161e88f33fd8 P 9d156a26f9e140dcb64782bed8729d55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:05:12.085484 9286 raft_consensus.cc:2229] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:05:12.086308 9286 raft_consensus.cc:2258] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:05:12.087582 9287 ts_tablet_manager.cc:1905] T 096573ae193e40c7b205161e88f33fd8 P 9d156a26f9e140dcb64782bed8729d55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:12.087733 9286 ts_tablet_manager.cc:1905] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:12.096241 9286 ts_tablet_manager.cc:1918] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:05:12.096436 9286 log.cc:1196] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/096573ae193e40c7b205161e88f33fd8
I20260928 02:05:12.096534 9285 tablet_replica.cc:330] T 096573ae193e40c7b205161e88f33fd8 P 5db2590c7c594cd9b59e518991ff1445: stopping tablet replica
I20260928 02:05:12.096735 9285 raft_consensus.cc:2229] T 096573ae193e40c7b205161e88f33fd8 P 5db2590c7c594cd9b59e518991ff1445 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:05:12.096964 9286 ts_tablet_manager.cc:1939] T 096573ae193e40c7b205161e88f33fd8 P 5e5d0dbc247b46bc8219f4db2a911ec7: Deleting consensus metadata
I20260928 02:05:12.097059 9285 raft_consensus.cc:2258] T 096573ae193e40c7b205161e88f33fd8 P 5db2590c7c594cd9b59e518991ff1445 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:05:12.098425 8682 catalog_manager.cc:4841] TS 5e5d0dbc247b46bc8219f4db2a911ec7 (127.1.182.130:39733): tablet 096573ae193e40c7b205161e88f33fd8 (table test-table [id=88908ec7eae846358747688ec0b02136]) successfully deleted
I20260928 02:05:12.099083 9285 ts_tablet_manager.cc:1905] T 096573ae193e40c7b205161e88f33fd8 P 5db2590c7c594cd9b59e518991ff1445: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:12.105554 9285 ts_tablet_manager.cc:1918] T 096573ae193e40c7b205161e88f33fd8 P 5db2590c7c594cd9b59e518991ff1445: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:05:12.105685 9285 log.cc:1196] T 096573ae193e40c7b205161e88f33fd8 P 5db2590c7c594cd9b59e518991ff1445: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/096573ae193e40c7b205161e88f33fd8
I20260928 02:05:12.095057 9287 ts_tablet_manager.cc:1918] T 096573ae193e40c7b205161e88f33fd8 P 9d156a26f9e140dcb64782bed8729d55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:05:12.105957 9287 log.cc:1196] T 096573ae193e40c7b205161e88f33fd8 P 9d156a26f9e140dcb64782bed8729d55: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561052730740-1754-0/minicluster-data/ts-0/wal/wals/096573ae193e40c7b205161e88f33fd8
I20260928 02:05:12.106472 9285 ts_tablet_manager.cc:1939] T 096573ae193e40c7b205161e88f33fd8 P 5db2590c7c594cd9b59e518991ff1445: Deleting consensus metadata
I20260928 02:05:12.106664 9287 ts_tablet_manager.cc:1939] T 096573ae193e40c7b205161e88f33fd8 P 9d156a26f9e140dcb64782bed8729d55: Deleting consensus metadata
I20260928 02:05:12.107713 8682 catalog_manager.cc:4841] TS 5db2590c7c594cd9b59e518991ff1445 (127.1.182.131:33133): tablet 096573ae193e40c7b205161e88f33fd8 (table test-table [id=88908ec7eae846358747688ec0b02136]) successfully deleted
I20260928 02:05:12.108018 8681 catalog_manager.cc:4841] TS 9d156a26f9e140dcb64782bed8729d55 (127.1.182.129:46607): tablet 096573ae193e40c7b205161e88f33fd8 (table test-table [id=88908ec7eae846358747688ec0b02136]) successfully deleted
I20260928 02:05:12.110394 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:12.079276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46120 (local address 127.1.182.129:46607)
0928 02:05:12.079510 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:12.079520 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:12.092936 (+ 13416us) server_negotiation.cc:407] Connection header received
0928 02:05:12.093454 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:12.093460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:12.093558 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:12.093639 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:12.095720 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:12.098302 (+ 2582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:12.099796 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:12.100192 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:12.100332 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:12.102946 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:12.102970 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:12.102985 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:12.103016 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:12.107057 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:12.107955 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:12.107964 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:12.107967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:12.108113 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:12.108598 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:12.108603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:12.108606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:12.109118 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:12.109425 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:12.109774 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 02:05:12.110004 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:05:12.111056 8877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:12.228410 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 8671
I20260928 02:05:12.252228 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 8894
I20260928 02:05:12.267565 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:12.253045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57306 (local address 127.1.182.130:39733)
0928 02:05:12.253220 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:12.253226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:12.253319 (+ 93us) server_negotiation.cc:407] Connection header received
0928 02:05:12.253476 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:12.253481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:12.253567 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:12.253674 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:12.254905 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:12.256186 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:12.257420 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:12.258119 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:12.258254 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:12.260854 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:12.260868 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:12.260908 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:12.260967 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:12.264834 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:12.265432 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:12.265442 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:12.265446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:12.265620 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:12.265962 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:12.265970 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:12.265975 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:12.266409 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:12.266702 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:12.266876 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 02:05:12.267180 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:05:12.268289 9017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:12.331645 9239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:12.331169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.0.0.1:55762)
0928 02:05:12.331353 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:12.331496 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 02:05:12.332677 9239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:12.332324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39651 (local address 127.0.0.1:55774)
0928 02:05:12.332478 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:12.332567 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 02:05:12.389721 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 8894
I20260928 02:05:12.409746 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 9035
I20260928 02:05:12.422565 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:12.410549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53600 (local address 127.1.182.131:33133)
0928 02:05:12.410731 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:12.410737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:12.410757 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:05:12.410951 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:12.410956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:12.411036 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:12.411180 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:12.412455 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:12.413197 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:12.414523 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:12.415026 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:12.415186 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:12.417586 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:12.417599 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:12.417616 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:12.417644 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:12.420274 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:12.420849 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:12.420855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:12.420858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:12.420988 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:12.421282 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:12.421285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:12.421287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:12.421610 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:12.421904 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:12.422166 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 02:05:12.422317 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:05:12.423271 9159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:12.536559 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 9035
I20260928 02:05:12.553650 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 8365
I20260928 02:05:12.554677 8532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:12.671897 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 8365
I20260928 02:05:12.692359 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 8466
I20260928 02:05:12.693547 8727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:12.814327 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 8466
2026-09-28T02:05:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9527 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:05:12 dist-test-slave-23bp krb5kdc[9296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:05:12 dist-test-slave-23bp krb5kdc[9296](info): set up 2 sockets
Sep 28 02:05:12 dist-test-slave-23bp krb5kdc[9296](info): commencing operation
krb5kdc: starting...
W20260928 02:05:13.918365 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:05:14 dist-test-slave-23bp krb5kdc[9296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:05:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:05:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.190.keytab.
Entry for principal oryx/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.190.keytab.
I20260928 02:05:14.133916 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:36451
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.190.keytab
--principal=oryx/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:36451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:14.453298 9309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:14.453619 9309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:14.453814 9309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:14.461100 9309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:05:14.461215 9309 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:05:14.461314 9309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:14.461486 9309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:14.461567 9309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:05:14.461661 9309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:05:14.471649 9309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:36451
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.190.keytab
--principal=oryx/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:36451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:14.472987 9309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:14.474325 9309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:14.486414 9314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:14.486495 9315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:14.486445 9317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:14.487138 9309 server_base.cc:1034] running on GCE node
I20260928 02:05:14.487811 9309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:14.489089 9309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:14.490301 9309 hybrid_clock.cc:648] HybridClock initialized: now 1790561114490251 us; error 29 us; skew 500 ppm
I20260928 02:05:14.490576 9309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:14 dist-test-slave-23bp krb5kdc[9296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561114, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:14.493770 9309 init.cc:377] Logged in from keytab as oryx/127.1.182.190@KRBTEST.COM (short username oryx)
I20260928 02:05:14.495190 9309 webserver.cc:460] Webserver started at http://127.1.182.190:46257/ using document root <none> and password file <none>
I20260928 02:05:14.495849 9309 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:14.495935 9309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:14.496200 9309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:14.498951 9309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "87050c7ca1a14e17982d9ffa888c5e43"
format_stamp: "Formatted at 2026-09-28 02:05:14 on dist-test-slave-23bp"
server_key: "9f0f6a1c8a9f8c6ac01a331e114f5035"
server_key_iv: "d48f71474768173c38d9ea18274e58ef"
server_key_version: "encryptionkey@0"
I20260928 02:05:14.499758 9309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "87050c7ca1a14e17982d9ffa888c5e43"
format_stamp: "Formatted at 2026-09-28 02:05:14 on dist-test-slave-23bp"
server_key: "9f0f6a1c8a9f8c6ac01a331e114f5035"
server_key_iv: "d48f71474768173c38d9ea18274e58ef"
server_key_version: "encryptionkey@0"
I20260928 02:05:14.504558 9309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:05:14.507817 9324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:14.509135 9309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:05:14.509268 9309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "87050c7ca1a14e17982d9ffa888c5e43"
format_stamp: "Formatted at 2026-09-28 02:05:14 on dist-test-slave-23bp"
server_key: "9f0f6a1c8a9f8c6ac01a331e114f5035"
server_key_iv: "d48f71474768173c38d9ea18274e58ef"
server_key_version: "encryptionkey@0"
I20260928 02:05:14.509446 9309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:14.532718 9309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:14.536662 9309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:14.536867 9309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:14.557569 9309 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:36451
I20260928 02:05:14.557617 9385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:36451 every 8 connection(s)
I20260928 02:05:14.558913 9309 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:05:14.562613 9386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:14.564354 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 9309
I20260928 02:05:14.564566 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:05:14.565387 1754 external_mini_cluster.cc:1428] Setting key b5254036a0b5a640ea3019343b657a1f
I20260928 02:05:14.574734 9386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43: Bootstrap starting.
I20260928 02:05:14.578406 9386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:14.579705 9386 log.cc:824] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:05:14 dist-test-slave-23bp krb5kdc[9296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM
I20260928 02:05:14.582593 9386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43: No bootstrap required, opened a new log
I20260928 02:05:14.588806 9389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:14.567585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47726 (local address 127.1.182.190:36451)
0928 02:05:14.568449 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:14.568464 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:14.568506 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:05:14.570179 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:14.570202 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:14.570722 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:14.571154 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:14.572826 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:14.574114 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:14.575841 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:14.576552 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:14.576685 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:14.581593 (+ 4908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:14.581601 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:14.581615 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:14.581653 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:14.584890 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:14.585598 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:14.585605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:14.585610 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:14.585772 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:14.586182 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:14.586191 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:14.586193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:14.586672 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:14.587007 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:14.587433 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 02:05:14.587825 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":423,"threads_started":1}
I20260928 02:05:14.590376 9386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87050c7ca1a14e17982d9ffa888c5e43" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36451 } }
I20260928 02:05:14.590744 9386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:14.590859 9386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87050c7ca1a14e17982d9ffa888c5e43, State: Initialized, Role: FOLLOWER
I20260928 02:05:14.591470 9386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87050c7ca1a14e17982d9ffa888c5e43" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36451 } }
I20260928 02:05:14.591745 9386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:05:14.591881 9386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:05:14.592067 9386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:14.594168 9386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87050c7ca1a14e17982d9ffa888c5e43" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36451 } }
I20260928 02:05:14.594641 9386 leader_election.cc:302] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87050c7ca1a14e17982d9ffa888c5e43; no voters:
I20260928 02:05:14.595345 9386 leader_election.cc:288] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:05:14.595449 9391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:05:14.596577 9391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43 [term 1 LEADER]: Becoming Leader. State: Replica: 87050c7ca1a14e17982d9ffa888c5e43, State: Running, Role: LEADER
I20260928 02:05:14.597085 9391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87050c7ca1a14e17982d9ffa888c5e43" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36451 } }
I20260928 02:05:14.597334 9386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:05:14.599764 9392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87050c7ca1a14e17982d9ffa888c5e43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87050c7ca1a14e17982d9ffa888c5e43" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36451 } } }
I20260928 02:05:14.600080 9392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:14.600903 9399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:05:14.600786 9393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87050c7ca1a14e17982d9ffa888c5e43. Latest consensus state: current_term: 1 leader_uuid: "87050c7ca1a14e17982d9ffa888c5e43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87050c7ca1a14e17982d9ffa888c5e43" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36451 } } }
I20260928 02:05:14.601094 9393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:14.606443 9399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:05:14.615604 9399 catalog_manager.cc:1347] Generated new cluster ID: b149620df73a4688a7875ba1a401ab9b
I20260928 02:05:14.615708 9399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:05:14.643901 9399 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:05:14.645462 9399 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:05:14.662604 9399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43: Generated new TSK 0
I20260928 02:05:14.663561 9399 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.182.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.182.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.182.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.129.keytab.
Entry for principal oryx/127.1.182.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.129.keytab.
I20260928 02:05:14.723726 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:36451
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.129.keytab
--principal=oryx/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:15.045567 9412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:15.045850 9412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:15.045953 9412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:15.053050 9412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:15.053155 9412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:15.053239 9412 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:05:15.053318 9412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:05:15.063313 9412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.129.keytab
--principal=oryx/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.182.190:36451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:15.064545 9412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:15.065778 9412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:15.081907 9418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:15.082394 9420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:15.082645 9417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:15.083374 9412 server_base.cc:1034] running on GCE node
I20260928 02:05:15.084020 9412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:15.084928 9412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:15.086170 9412 hybrid_clock.cc:648] HybridClock initialized: now 1790561115086109 us; error 39 us; skew 500 ppm
I20260928 02:05:15.086467 9412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:15 dist-test-slave-23bp krb5kdc[9296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561115, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:15.089481 9412 init.cc:377] Logged in from keytab as oryx/127.1.182.129@KRBTEST.COM (short username oryx)
I20260928 02:05:15.090953 9412 webserver.cc:460] Webserver started at http://127.1.182.129:44377/ using document root <none> and password file <none>
I20260928 02:05:15.091682 9412 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:15.091811 9412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:15.092123 9412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:15.094935 9412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "ecd8b31074bd483e889ac06dee87dfe8"
format_stamp: "Formatted at 2026-09-28 02:05:15 on dist-test-slave-23bp"
server_key: "7a1dd0b7d96b41fe401de5f6b7f2fd68"
server_key_iv: "0e71de29b0348b3182dfcc10277fe792"
server_key_version: "encryptionkey@0"
I20260928 02:05:15.095827 9412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "ecd8b31074bd483e889ac06dee87dfe8"
format_stamp: "Formatted at 2026-09-28 02:05:15 on dist-test-slave-23bp"
server_key: "7a1dd0b7d96b41fe401de5f6b7f2fd68"
server_key_iv: "0e71de29b0348b3182dfcc10277fe792"
server_key_version: "encryptionkey@0"
I20260928 02:05:15.100966 9412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 02:05:15.104477 9427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:15.105737 9412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:05:15.105906 9412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "ecd8b31074bd483e889ac06dee87dfe8"
format_stamp: "Formatted at 2026-09-28 02:05:15 on dist-test-slave-23bp"
server_key: "7a1dd0b7d96b41fe401de5f6b7f2fd68"
server_key_iv: "0e71de29b0348b3182dfcc10277fe792"
server_key_version: "encryptionkey@0"
I20260928 02:05:15.106122 9412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:15.119586 9412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:15.123440 9412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:15.123725 9412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:15.126248 9412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:15.126355 9412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:15.126513 9412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:15.126588 9412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:05:15 dist-test-slave-23bp krb5kdc[9296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561115, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM
I20260928 02:05:15.148113 9436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:15.127013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:51511 (local address 127.1.182.190:36451)
0928 02:05:15.127535 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:15.127541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:15.128742 (+ 1201us) server_negotiation.cc:407] Connection header received
0928 02:05:15.130745 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:15.130751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:15.130845 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:15.130974 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:15.133189 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:15.134294 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:15.136911 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:15.137337 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:15.137482 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:15.141830 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:15.141835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:15.141838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:15.141872 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:15.144974 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:15.145721 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:15.145727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:15.145731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:15.145852 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:15.146290 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:15.146295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:15.146299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:15.146580 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:15.146740 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:15.147520 (+ 780us) server_negotiation.cc:299] Negotiation successful
0928 02:05:15.147740 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":262,"threads_started":1}
I20260928 02:05:15.149204 9437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:15.127667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:36451 (local address 127.1.182.129:51511)
0928 02:05:15.128550 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:15.128599 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:15.130457 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:15.131255 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:15.131264 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:15.132015 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:15.132991 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:15.133019 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:15.134455 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:15.134462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:15.136758 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:15.136771 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:15.137574 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:15.137582 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:15.137805 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:15.138514 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:15.138536 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:15.141592 (+ 3056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:15.145125 (+ 3533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:15.145134 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:15.145152 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:15.145593 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:15.145960 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:15.145967 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:15.145973 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:15.146172 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:15.146708 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:15.146717 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:15.147140 (+ 423us) client_negotiation.cc:769] Sending connection context
0928 02:05:15.147435 (+ 295us) client_negotiation.cc:240] Negotiation successful
0928 02:05:15.147989 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":372,"threads_started":1}
I20260928 02:05:15.172235 9436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:15.155568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:50277 (local address 127.1.182.190:36451)
0928 02:05:15.155713 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:15.155717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:15.156463 (+ 746us) server_negotiation.cc:407] Connection header received
0928 02:05:15.156665 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:15.156670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:15.156760 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:15.156867 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:15.158266 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:15.159227 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:15.161907 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:15.162664 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:15.162835 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:15.166027 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:15.166036 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:15.166043 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:15.166083 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:15.169145 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:15.169976 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:15.169979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:15.169982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:15.170099 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:15.170595 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:15.170601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:15.170607 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:15.171030 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:15.171233 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:15.171666 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 02:05:15.171890 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:05:15.172374 9487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:15.155737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:36451 (local address 127.1.182.129:50277)
0928 02:05:15.156352 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:15.156367 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:15.156527 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:15.156926 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:15.156934 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:15.157628 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:15.158108 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:15.158125 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:15.159401 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:15.159411 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:15.161676 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:15.161700 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:15.163161 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:15.163170 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:15.163462 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:15.164375 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:15.164400 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:15.165823 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:15.169259 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:15.169264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:15.169268 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:15.169844 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:15.170211 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:15.170219 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:15.170230 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:15.170426 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:15.171190 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:15.171202 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:15.171412 (+ 210us) client_negotiation.cc:769] Sending connection context
0928 02:05:15.171579 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 02:05:15.171919 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":343,"threads_started":1}
I20260928 02:05:15.181340 9412 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:35749
I20260928 02:05:15.181435 9547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:35749 every 8 connection(s)
I20260928 02:05:15.182571 9412 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:05:15.184599 9548 heartbeater.cc:347] Connected to a master server at 127.1.182.190:36451
I20260928 02:05:15.184823 9548 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:15.185051 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 9412
I20260928 02:05:15.185248 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:05:15.185550 9548 heartbeater.cc:511] Master 127.1.182.190:36451 requested a full tablet report, sending...
I20260928 02:05:15.185907 1754 external_mini_cluster.cc:1428] Setting key 5037fa9df3416bd46a37cfdc9dd8d742
I20260928 02:05:15.187772 9341 ts_manager.cc:194] Registered new tserver with Master: ecd8b31074bd483e889ac06dee87dfe8 (127.1.182.129:35749)
I20260928 02:05:15.189138 9341 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:51511
WARNING: no policy specified for oryx/127.1.182.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.182.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.182.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.130.keytab.
Entry for principal oryx/127.1.182.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.130.keytab.
I20260928 02:05:15.232347 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:36451
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.130.keytab
--principal=oryx/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:15.552744 9552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:15.553063 9552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:15.553231 9552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:15.560736 9552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:15.560883 9552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:15.561023 9552 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:05:15.561158 9552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:05:15.571372 9552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.130.keytab
--principal=oryx/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.182.190:36451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:15.572652 9552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:15.573969 9552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:15.586951 9560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:15.586933 9558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:15.587389 9557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:15.587908 9552 server_base.cc:1034] running on GCE node
I20260928 02:05:15.588418 9552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:15.589237 9552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:15.590472 9552 hybrid_clock.cc:648] HybridClock initialized: now 1790561115590404 us; error 50 us; skew 500 ppm
I20260928 02:05:15.590796 9552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:15 dist-test-slave-23bp krb5kdc[9296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561115, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:15.593899 9552 init.cc:377] Logged in from keytab as oryx/127.1.182.130@KRBTEST.COM (short username oryx)
I20260928 02:05:15.595212 9552 webserver.cc:460] Webserver started at http://127.1.182.130:42597/ using document root <none> and password file <none>
I20260928 02:05:15.595870 9552 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:15.595952 9552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:15.596189 9552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:15.599366 9552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "9cc0277329224d2fb3a3aee237f4b6b5"
format_stamp: "Formatted at 2026-09-28 02:05:15 on dist-test-slave-23bp"
server_key: "eb2514eb2b080245c6b8ee5ac6f098ed"
server_key_iv: "644bf3d91f5b2c2dff7161d62daae246"
server_key_version: "encryptionkey@0"
I20260928 02:05:15.600114 9552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "9cc0277329224d2fb3a3aee237f4b6b5"
format_stamp: "Formatted at 2026-09-28 02:05:15 on dist-test-slave-23bp"
server_key: "eb2514eb2b080245c6b8ee5ac6f098ed"
server_key_iv: "644bf3d91f5b2c2dff7161d62daae246"
server_key_version: "encryptionkey@0"
I20260928 02:05:15.605445 9552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:05:15.608697 9567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:15.610028 9552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:05:15.610149 9552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "9cc0277329224d2fb3a3aee237f4b6b5"
format_stamp: "Formatted at 2026-09-28 02:05:15 on dist-test-slave-23bp"
server_key: "eb2514eb2b080245c6b8ee5ac6f098ed"
server_key_iv: "644bf3d91f5b2c2dff7161d62daae246"
server_key_version: "encryptionkey@0"
I20260928 02:05:15.610366 9552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:15.631690 9552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:15.635599 9552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:15.635864 9552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:15.638377 9552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:15.638496 9552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:15.638644 9552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:15.638718 9552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:05:15 dist-test-slave-23bp krb5kdc[9296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561115, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM
I20260928 02:05:15.661355 9436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:15.639158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:37461 (local address 127.1.182.190:36451)
0928 02:05:15.639332 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:15.639338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:15.640750 (+ 1412us) server_negotiation.cc:407] Connection header received
0928 02:05:15.643988 (+ 3238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:15.643997 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:15.644097 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:15.644216 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:15.646862 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:15.647857 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:15.649476 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:15.649882 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:15.650018 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:15.654361 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:15.654370 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:15.654377 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:15.654417 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:15.658124 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:15.659055 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:15.659061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:15.659088 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:15.659247 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:15.659733 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:15.659740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:15.659745 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:15.660002 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:15.660143 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:15.660801 (+ 658us) server_negotiation.cc:299] Negotiation successful
0928 02:05:15.661009 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:05:15.662964 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:15.639676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:36451 (local address 127.1.182.130:37461)
0928 02:05:15.640554 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:15.640608 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:15.643612 (+ 3004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:15.644562 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:15.644582 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:15.645548 (+ 966us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:15.646657 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:15.646687 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:15.648024 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:15.648032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:15.649313 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:15.649324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:15.650122 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:15.650130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:15.650443 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:15.651221 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:15.651266 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:15.654118 (+ 2852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:15.658296 (+ 4178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:15.658302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:15.658314 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:15.658928 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:15.659396 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:15.659401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:15.659411 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:15.659608 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:15.660134 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:15.660142 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:15.660466 (+ 324us) client_negotiation.cc:769] Sending connection context
0928 02:05:15.660725 (+ 259us) client_negotiation.cc:240] Negotiation successful
0928 02:05:15.661221 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":328,"threads_started":1}
I20260928 02:05:15.686478 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:15.669054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:36451 (local address 127.1.182.130:56641)
0928 02:05:15.669796 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:15.669821 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:15.669994 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:15.670323 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:15.670331 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:15.671095 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:15.671610 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:15.671626 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:15.673060 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:15.673069 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:15.675400 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:15.675418 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:15.676406 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:15.676413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:15.676572 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:15.677416 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:15.677444 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:15.678254 (+ 810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:15.682460 (+ 4206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:15.682469 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:15.682475 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:15.683111 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:15.683648 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:15.683656 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:15.683662 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:15.683845 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:15.684863 (+ 1018us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:15.684871 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:15.685139 (+ 268us) client_negotiation.cc:769] Sending connection context
0928 02:05:15.685319 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 02:05:15.685742 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":400,"threads_started":1}
I20260928 02:05:15.687530 9436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:15.669055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:56641 (local address 127.1.182.190:36451)
0928 02:05:15.669276 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:15.669284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:15.669916 (+ 632us) server_negotiation.cc:407] Connection header received
0928 02:05:15.670128 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:15.670132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:15.670186 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:15.670262 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:15.671802 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:15.672898 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:15.675565 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:15.676007 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:15.676163 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:15.678472 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:15.678478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:15.678485 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:15.678531 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:15.682306 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:15.683286 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:15.683295 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:15.683301 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:15.683487 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:15.684010 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:15.684019 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:15.684024 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:15.684564 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:15.684787 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:15.686519 (+ 1732us) server_negotiation.cc:299] Negotiation successful
0928 02:05:15.687118 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:05:15.697005 9552 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:43485
I20260928 02:05:15.697098 9686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:43485 every 8 connection(s)
I20260928 02:05:15.698308 9552 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:05:15.700598 9687 heartbeater.cc:347] Connected to a master server at 127.1.182.190:36451
I20260928 02:05:15.700800 9687 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:15.701443 9687 heartbeater.cc:511] Master 127.1.182.190:36451 requested a full tablet report, sending...
I20260928 02:05:15.702814 9341 ts_manager.cc:194] Registered new tserver with Master: 9cc0277329224d2fb3a3aee237f4b6b5 (127.1.182.130:43485)
I20260928 02:05:15.703737 9341 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:37461
I20260928 02:05:15.704025 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 9552
I20260928 02:05:15.704218 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:05:15.704810 1754 external_mini_cluster.cc:1428] Setting key c10f3ec10122286fec92c470ecdab2c7
WARNING: no policy specified for oryx/127.1.182.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.182.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.182.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.131.keytab.
Entry for principal oryx/127.1.182.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.131.keytab.
I20260928 02:05:15.751299 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:36451
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.131.keytab
--principal=oryx/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:16.070511 9691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:16.070784 9691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:16.070940 9691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:16.078090 9691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:16.078220 9691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:16.078368 9691 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:05:16.078491 9691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:05:16.088414 9691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.131.keytab
--principal=oryx/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.182.190:36451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:16.089771 9691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:16.091135 9691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:16.104014 9696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:16.104065 9699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:16.104086 9697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:16.105118 9691 server_base.cc:1034] running on GCE node
I20260928 02:05:16.105705 9691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:16.106637 9691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:16.107868 9691 hybrid_clock.cc:648] HybridClock initialized: now 1790561116107802 us; error 47 us; skew 500 ppm
I20260928 02:05:16.108173 9691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:16 dist-test-slave-23bp krb5kdc[9296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561116, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:16.111363 9691 init.cc:377] Logged in from keytab as oryx/127.1.182.131@KRBTEST.COM (short username oryx)
I20260928 02:05:16.112809 9691 webserver.cc:460] Webserver started at http://127.1.182.131:35793/ using document root <none> and password file <none>
I20260928 02:05:16.113641 9691 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:16.113770 9691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:16.114087 9691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:16.117194 9691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "417e8867b6b74f14a2402ef52dd71ad6"
format_stamp: "Formatted at 2026-09-28 02:05:16 on dist-test-slave-23bp"
server_key: "ff65c5a6e6809ef4327b822627f024f8"
server_key_iv: "94b565a9b9f32736f33ea472057cee39"
server_key_version: "encryptionkey@0"
I20260928 02:05:16.117981 9691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "417e8867b6b74f14a2402ef52dd71ad6"
format_stamp: "Formatted at 2026-09-28 02:05:16 on dist-test-slave-23bp"
server_key: "ff65c5a6e6809ef4327b822627f024f8"
server_key_iv: "94b565a9b9f32736f33ea472057cee39"
server_key_version: "encryptionkey@0"
I20260928 02:05:16.122800 9691 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260928 02:05:16.126026 9706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:16.127368 9691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:05:16.127540 9691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "417e8867b6b74f14a2402ef52dd71ad6"
format_stamp: "Formatted at 2026-09-28 02:05:16 on dist-test-slave-23bp"
server_key: "ff65c5a6e6809ef4327b822627f024f8"
server_key_iv: "94b565a9b9f32736f33ea472057cee39"
server_key_version: "encryptionkey@0"
I20260928 02:05:16.127758 9691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:16.149693 9691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:16.153755 9691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:16.154024 9691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:16.156678 9691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:16.156782 9691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:16.156915 9691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:16.157004 9691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:05:16 dist-test-slave-23bp krb5kdc[9296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561116, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM
I20260928 02:05:16.181633 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:16.157595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:36451 (local address 127.1.182.131:33201)
0928 02:05:16.158439 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:16.158487 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:16.160435 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:16.161213 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:16.161232 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:16.162024 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:16.162867 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:16.162886 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:16.164240 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:16.164246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:16.166111 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:16.166126 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:16.167301 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:16.167319 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:16.167699 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:16.168504 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:16.168547 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:16.172509 (+ 3962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:16.176439 (+ 3930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:16.176453 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:16.176471 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:16.177284 (+ 813us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:16.177739 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:16.177744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:16.177748 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:16.177958 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:16.178618 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:16.178630 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:16.179182 (+ 552us) client_negotiation.cc:769] Sending connection context
0928 02:05:16.179563 (+ 381us) client_negotiation.cc:240] Negotiation successful
0928 02:05:16.180167 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":371,"threads_started":1}
I20260928 02:05:16.182713 9436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:16.157041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:33201 (local address 127.1.182.190:36451)
0928 02:05:16.157221 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:16.157227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:16.158630 (+ 1403us) server_negotiation.cc:407] Connection header received
0928 02:05:16.160734 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:16.160740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:16.160826 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:16.160934 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:16.163043 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:16.164088 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:16.166270 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:16.166869 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:16.167037 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:16.172800 (+ 5763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:16.172809 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:16.172816 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:16.172895 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:16.176274 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:16.177430 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:16.177438 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:16.177442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:16.177607 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:16.178089 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:16.178095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:16.178101 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:16.178489 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:16.178639 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:16.182210 (+ 3571us) server_negotiation.cc:299] Negotiation successful
0928 02:05:16.182438 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:05:16.191932 9548 heartbeater.cc:503] Master 127.1.182.190:36451 was elected leader, sending a full tablet report...
I20260928 02:05:16.206713 9436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:16.188178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:59611 (local address 127.1.182.190:36451)
0928 02:05:16.188385 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:16.188394 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:16.191803 (+ 3409us) server_negotiation.cc:407] Connection header received
0928 02:05:16.192046 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:16.192052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:16.192134 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:16.192231 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:16.194675 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:16.195552 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:16.198225 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:16.198907 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:16.199056 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:16.201290 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:16.201295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:16.201298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:16.201390 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:16.204170 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:16.204836 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:16.204843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:16.204851 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:16.204962 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:16.205319 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:16.205324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:16.205327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:16.205594 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:16.205751 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:16.206131 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 02:05:16.206339 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:05:16.206779 9780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:16.188177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:36451 (local address 127.1.182.131:59611)
0928 02:05:16.191664 (+ 3487us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:16.191697 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:16.191864 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:16.192268 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:16.192275 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:16.194022 (+ 1747us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:16.194483 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:16.194500 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:16.195758 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:16.195763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:16.198038 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:16.198059 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:16.199175 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:16.199185 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:16.199455 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:16.200333 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:16.200350 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:16.201092 (+ 742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:16.204306 (+ 3214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:16.204311 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:16.204315 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:16.204714 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:16.205069 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:16.205075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:16.205079 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:16.205206 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:16.205711 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:16.205715 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:16.205890 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 02:05:16.206035 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 02:05:16.206349 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3290,"thread_start_us":3127,"threads_started":1}
I20260928 02:05:16.220705 9691 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:33033
I20260928 02:05:16.220732 9825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:33033 every 8 connection(s)
I20260928 02:05:16.221941 9691 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:05:16.223908 9826 heartbeater.cc:347] Connected to a master server at 127.1.182.190:36451
I20260928 02:05:16.224149 9826 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:16.224162 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 9691
I20260928 02:05:16.224329 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:05:16.224761 1754 external_mini_cluster.cc:1428] Setting key d54fef8cccaab4de1851a80c0dda0ed2
I20260928 02:05:16.224822 9826 heartbeater.cc:511] Master 127.1.182.190:36451 requested a full tablet report, sending...
I20260928 02:05:16.226239 9341 ts_manager.cc:194] Registered new tserver with Master: 417e8867b6b74f14a2402ef52dd71ad6 (127.1.182.131:33033)
I20260928 02:05:16.227142 9341 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:33201
I20260928 02:05:16.227804 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:05:16 dist-test-slave-23bp krb5kdc[9296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561116, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:05:16 dist-test-slave-23bp krb5kdc[9296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:16 dist-test-slave-23bp krb5kdc[9296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:16.706616 9687 heartbeater.cc:503] Master 127.1.182.190:36451 was elected leader, sending a full tablet report...
I20260928 02:05:17.229588 9826 heartbeater.cc:503] Master 127.1.182.190:36451 was elected leader, sending a full tablet report...
W20260928 02:05:17.340692 9834 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.070s user 0.007s sys 0.007s
W20260928 02:05:17.341743 9436 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:16.265720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47742 (local address 127.1.182.190:36451)
0928 02:05:16.265969 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:16.265977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:16.266257 (+ 280us) server_negotiation.cc:407] Connection header received
0928 02:05:16.266401 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:16.266405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:16.266468 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:16.266553 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:16.267848 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:16.268567 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:16.269723 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:16.270121 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:16.270210 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:17.341530 (+1071320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47742: BlockingRecv error: recv got EOF from 127.0.0.1:47742 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 28 02:05:17 dist-test-slave-23bp krb5kdc[9296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561116, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM
I20260928 02:05:17.364163 9436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:17.347675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.1.182.190:36451)
0928 02:05:17.347897 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:17.347903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:17.348074 (+ 171us) server_negotiation.cc:407] Connection header received
0928 02:05:17.348205 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:17.348208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:17.348311 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:17.348396 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:17.349586 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.350291 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.351673 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.352123 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.352216 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:17.359267 (+ 7051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:17.359271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:17.359274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:17.359309 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:17.361862 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:17.362466 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:17.362470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:17.362473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:17.362582 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:17.362877 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:17.362880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:17.362882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:17.363214 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:17.363427 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:17.363659 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 02:05:17.363820 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:05:17.368242 9341 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:05:17.374970 9341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:05:17.398959 9845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:17.386049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:43485 (local address 127.0.0.1:50000)
0928 02:05:17.386955 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:17.387000 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:17.387258 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:17.388472 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:17.388481 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:17.388510 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:17.388974 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:17.388994 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.391768 (+ 2774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.391777 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:17.394910 (+ 3133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:17.394930 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.397572 (+ 2642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.397582 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:17.397927 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:17.398014 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 02:05:17.398162 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 02:05:17.398410 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":383,"threads_started":1}
I20260928 02:05:17.400794 9846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:17.386204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50000 (local address 127.1.182.130:43485)
0928 02:05:17.387827 (+ 1623us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:17.387838 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:17.387870 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:05:17.387994 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:17.388005 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:17.388252 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:17.389249 (+ 997us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:17.389502 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.391580 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.395239 (+ 3659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.397274 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.399796 (+ 2522us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:17.400096 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:17.400189 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 02:05:17.400408 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1473,"thread_start_us":1353,"threads_started":1}
I20260928 02:05:17.405149 9617 tablet_service.cc:1467] Processing CreateTablet for tablet ffafda5910c84eabb9699a445de16745 (DEFAULT_TABLE table=test-table [id=a44e04f77b0344d1ad66e62aca77b304]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:05:17.406520 9617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffafda5910c84eabb9699a445de16745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:17.408473 9845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:17.388282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:35749 (local address 127.0.0.1:50460)
0928 02:05:17.399275 (+ 10993us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:17.399298 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:17.399418 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:17.402134 (+ 2716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:17.402138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:17.402161 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:17.402329 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:17.402341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.404165 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.404169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:17.405985 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:17.405998 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.407733 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.407741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:17.407914 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:17.407947 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:05:17.408047 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 02:05:17.408181 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10864,"thread_start_us":12793,"threads_started":1}
I20260928 02:05:17.408571 9850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:17.388736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.1.182.129:35749)
0928 02:05:17.401658 (+ 12922us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:17.401664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:17.401689 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:05:17.401762 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:17.401767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:17.401954 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:17.402156 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:17.402479 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.403998 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.406151 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.407571 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.407723 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:17.407904 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:17.408101 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 02:05:17.408254 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12788,"thread_start_us":12598,"threads_started":1}
I20260928 02:05:17.410094 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:17.387241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:33033 (local address 127.0.0.1:49008)
0928 02:05:17.388141 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:17.388157 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:17.388258 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:17.403423 (+ 15165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:17.403429 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:17.403464 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:17.403838 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:17.403848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.405848 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.405855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:17.407406 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:17.407417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.409364 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.409372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:17.409548 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:17.409580 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:05:17.409645 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:05:17.409761 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":656,"threads_started":1}
I20260928 02:05:17.410861 9848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:17.387663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49008 (local address 127.1.182.131:33033)
0928 02:05:17.402846 (+ 15183us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:17.402856 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:17.402884 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:05:17.402973 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:17.402980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:17.403225 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:17.404027 (+ 802us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:17.404230 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.405689 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.407554 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.409146 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.410192 (+ 1046us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:17.410391 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:17.410478 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 02:05:17.410601 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15023,"thread_start_us":15449,"threads_started":1}
I20260928 02:05:17.412333 9478 tablet_service.cc:1467] Processing CreateTablet for tablet ffafda5910c84eabb9699a445de16745 (DEFAULT_TABLE table=test-table [id=a44e04f77b0344d1ad66e62aca77b304]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:05:17.413863 9478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffafda5910c84eabb9699a445de16745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:17.413811 9756 tablet_service.cc:1467] Processing CreateTablet for tablet ffafda5910c84eabb9699a445de16745 (DEFAULT_TABLE table=test-table [id=a44e04f77b0344d1ad66e62aca77b304]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:05:17.415050 9756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffafda5910c84eabb9699a445de16745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:17.423748 9851 tablet_bootstrap.cc:492] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5: Bootstrap starting.
I20260928 02:05:17.427785 9852 tablet_bootstrap.cc:492] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6: Bootstrap starting.
I20260928 02:05:17.428781 9851 tablet_bootstrap.cc:654] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:17.429863 9851 log.cc:824] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:05:17.431955 9853 tablet_bootstrap.cc:492] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8: Bootstrap starting.
I20260928 02:05:17.432130 9852 tablet_bootstrap.cc:654] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:17.433542 9852 log.cc:824] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:05:17.435570 9851 tablet_bootstrap.cc:492] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5: No bootstrap required, opened a new log
I20260928 02:05:17.435843 9853 tablet_bootstrap.cc:654] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:17.435876 9851 ts_tablet_manager.cc:1397] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20260928 02:05:17.436949 9852 tablet_bootstrap.cc:492] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6: No bootstrap required, opened a new log
I20260928 02:05:17.437196 9852 ts_tablet_manager.cc:1397] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 02:05:17.437260 9853 log.cc:824] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:05:17.440400 9853 tablet_bootstrap.cc:492] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8: No bootstrap required, opened a new log
I20260928 02:05:17.440678 9853 ts_tablet_manager.cc:1397] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260928 02:05:17.444576 9852 raft_consensus.cc:348] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } }
I20260928 02:05:17.444944 9852 raft_consensus.cc:374] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:17.445080 9852 raft_consensus.cc:729] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417e8867b6b74f14a2402ef52dd71ad6, State: Initialized, Role: FOLLOWER
I20260928 02:05:17.445708 9852 consensus_queue.cc:260] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } }
I20260928 02:05:17.447435 9852 ts_tablet_manager.cc:1428] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6: Time spent starting tablet: real 0.010s user 0.005s sys 0.005s
I20260928 02:05:17.447450 9851 raft_consensus.cc:348] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } }
I20260928 02:05:17.447912 9851 raft_consensus.cc:374] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:17.448024 9851 raft_consensus.cc:729] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc0277329224d2fb3a3aee237f4b6b5, State: Initialized, Role: FOLLOWER
I20260928 02:05:17.448632 9851 consensus_queue.cc:260] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } }
I20260928 02:05:17.450415 9851 ts_tablet_manager.cc:1428] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
W20260928 02:05:17.451202 9688 tablet.cc:2369] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:05:17.453962 9853 raft_consensus.cc:348] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } }
I20260928 02:05:17.454367 9853 raft_consensus.cc:374] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:17.454521 9853 raft_consensus.cc:729] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecd8b31074bd483e889ac06dee87dfe8, State: Initialized, Role: FOLLOWER
I20260928 02:05:17.455116 9853 consensus_queue.cc:260] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } }
I20260928 02:05:17.456745 9853 ts_tablet_manager.cc:1428] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
W20260928 02:05:17.474517 9827 tablet.cc:2369] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:05:17.685861 9549 tablet.cc:2369] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:05:17.730381 9859 raft_consensus.cc:482] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:17.730666 9859 raft_consensus.cc:504] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } }
I20260928 02:05:17.732479 9859 leader_election.cc:288] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cc0277329224d2fb3a3aee237f4b6b5 (127.1.182.130:43485), 417e8867b6b74f14a2402ef52dd71ad6 (127.1.182.131:33033)
I20260928 02:05:17.741046 9848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:17.733868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:36163 (local address 127.1.182.131:33033)
0928 02:05:17.734086 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:17.734092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:17.734227 (+ 135us) server_negotiation.cc:407] Connection header received
0928 02:05:17.734381 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:17.734387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:17.734465 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:17.734562 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:17.735260 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.736273 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.738770 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.739920 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.740121 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:17.740206 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:17.740640 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 02:05:17.740777 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:05:17.741407 9846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:17.732772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:42503 (local address 127.1.182.130:43485)
0928 02:05:17.732984 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:17.732992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:17.733668 (+ 676us) server_negotiation.cc:407] Connection header received
0928 02:05:17.734007 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:17.734014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:17.734098 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:17.734205 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:17.735112 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.736365 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.739010 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.740339 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.740506 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:17.740577 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:17.740944 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 02:05:17.741044 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:05:17.741403 9860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:17.732787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:43485 (local address 127.1.182.129:42503)
0928 02:05:17.733391 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:17.733406 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:17.733741 (+ 335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:17.734430 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:17.734436 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:17.734472 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:17.734918 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:17.734932 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.736541 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.736547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:17.738816 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:17.738831 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.740612 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.740619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:17.740789 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:17.740819 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 02:05:17.740905 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:05:17.741015 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":27,"thread_start_us":324,"threads_started":1}
I20260928 02:05:17.741734 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:17.733517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:33033 (local address 127.1.182.129:36163)
0928 02:05:17.734124 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:17.734137 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:17.734240 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:17.734591 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:17.734594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:17.734624 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:17.735027 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:17.735035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.736430 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.736436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:17.738606 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:17.738619 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.740172 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.740205 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:17.740433 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:17.740469 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:05:17.741213 (+ 744us) client_negotiation.cc:240] Negotiation successful
0928 02:05:17.741342 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":1,"thread_start_us":343,"threads_started":1}
I20260928 02:05:17.742534 9642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffafda5910c84eabb9699a445de16745" candidate_uuid: "ecd8b31074bd483e889ac06dee87dfe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" is_pre_election: true
I20260928 02:05:17.742877 9781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffafda5910c84eabb9699a445de16745" candidate_uuid: "ecd8b31074bd483e889ac06dee87dfe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "417e8867b6b74f14a2402ef52dd71ad6" is_pre_election: true
I20260928 02:05:17.743105 9642 raft_consensus.cc:2454] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecd8b31074bd483e889ac06dee87dfe8 in term 0.
I20260928 02:05:17.743494 9781 raft_consensus.cc:2454] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecd8b31074bd483e889ac06dee87dfe8 in term 0.
I20260928 02:05:17.743789 9428 leader_election.cc:302] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cc0277329224d2fb3a3aee237f4b6b5, ecd8b31074bd483e889ac06dee87dfe8; no voters:
I20260928 02:05:17.744242 9859 raft_consensus.cc:2781] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:05:17.744385 9859 raft_consensus.cc:482] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:05:17.744493 9859 raft_consensus.cc:3037] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:17.746598 9859 raft_consensus.cc:504] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } }
I20260928 02:05:17.747284 9859 leader_election.cc:288] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 [CANDIDATE]: Term 1 election: Requested vote from peers 9cc0277329224d2fb3a3aee237f4b6b5 (127.1.182.130:43485), 417e8867b6b74f14a2402ef52dd71ad6 (127.1.182.131:33033)
I20260928 02:05:17.747710 9642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffafda5910c84eabb9699a445de16745" candidate_uuid: "ecd8b31074bd483e889ac06dee87dfe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc0277329224d2fb3a3aee237f4b6b5"
I20260928 02:05:17.747795 9781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffafda5910c84eabb9699a445de16745" candidate_uuid: "ecd8b31074bd483e889ac06dee87dfe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "417e8867b6b74f14a2402ef52dd71ad6"
I20260928 02:05:17.747906 9642 raft_consensus.cc:3037] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:17.748023 9781 raft_consensus.cc:3037] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:17.750048 9642 raft_consensus.cc:2454] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecd8b31074bd483e889ac06dee87dfe8 in term 1.
I20260928 02:05:17.750190 9781 raft_consensus.cc:2454] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecd8b31074bd483e889ac06dee87dfe8 in term 1.
I20260928 02:05:17.750413 9428 leader_election.cc:302] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cc0277329224d2fb3a3aee237f4b6b5, ecd8b31074bd483e889ac06dee87dfe8; no voters:
I20260928 02:05:17.750702 9859 raft_consensus.cc:2781] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:05:17.751256 9859 raft_consensus.cc:686] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 [term 1 LEADER]: Becoming Leader. State: Replica: ecd8b31074bd483e889ac06dee87dfe8, State: Running, Role: LEADER
I20260928 02:05:17.752244 9859 consensus_queue.cc:237] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } }
I20260928 02:05:17.756412 9340 catalog_manager.cc:5495] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecd8b31074bd483e889ac06dee87dfe8 (127.1.182.129). New cstate: current_term: 1 leader_uuid: "ecd8b31074bd483e889ac06dee87dfe8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } health_report { overall_health: HEALTHY } } }
I20260928 02:05:17.852224 9850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:17.846867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50476 (local address 127.1.182.129:35749)
0928 02:05:17.847129 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:17.847135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:17.847232 (+ 97us) server_negotiation.cc:407] Connection header received
0928 02:05:17.847521 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:17.847525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:17.847597 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:17.847752 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:05:17.848100 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.848781 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.850079 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:17.850472 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:17.850631 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:17.851135 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:05:17.851655 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:05:17.851849 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:17.851896 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 02:05:17.851981 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 02:05:17.859869 9781 raft_consensus.cc:1261] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6 [term 1 FOLLOWER]: Refusing update from remote peer ecd8b31074bd483e889ac06dee87dfe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:05:17.859870 9642 raft_consensus.cc:1261] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 1 FOLLOWER]: Refusing update from remote peer ecd8b31074bd483e889ac06dee87dfe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:05:17.860795 9862 consensus_queue.cc:1035] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:05:17.861135 9859 consensus_queue.cc:1035] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:05:17.872268 9865 mvcc.cc:205] Tried to move back new op lower bound from 7334138338745110528 to 7334138338316550144. Current Snapshot: MvccSnapshot[applied={T|T < 7334138338745110528}]
I20260928 02:05:17.874400 9866 mvcc.cc:205] Tried to move back new op lower bound from 7334138338745110528 to 7334138338316550144. Current Snapshot: MvccSnapshot[applied={T|T < 7334138338745110528}]
I20260928 02:05:17.885664 9341 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:05:17.885926 9341 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:05:17.889084 9341 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43: Sending DeleteTablet for 3 replicas of tablet ffafda5910c84eabb9699a445de16745
I20260928 02:05:17.890156 9617 tablet_service.cc:1514] Processing DeleteTablet for tablet ffafda5910c84eabb9699a445de16745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:05:17 UTC) from {username='oryx', principal='oryx/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:50000
I20260928 02:05:17.890311 9756 tablet_service.cc:1514] Processing DeleteTablet for tablet ffafda5910c84eabb9699a445de16745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:05:17 UTC) from {username='oryx', principal='oryx/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:49008
I20260928 02:05:17.891011 9478 tablet_service.cc:1514] Processing DeleteTablet for tablet ffafda5910c84eabb9699a445de16745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:05:17 UTC) from {username='oryx', principal='oryx/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:50460
I20260928 02:05:17.891206 9876 tablet_replica.cc:330] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6: stopping tablet replica
I20260928 02:05:17.891607 9876 raft_consensus.cc:2229] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:05:17.892131 9876 raft_consensus.cc:2258] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:05:17.892765 9340 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:05:17.893426 9340 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:05:17.894245 9876 ts_tablet_manager.cc:1905] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:17.900955 9876 ts_tablet_manager.cc:1918] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:05:17.901726 9616 tablet_service.cc:1467] Processing CreateTablet for tablet 8193071af6644bf69dc08024cbd6c37f (DEFAULT_TABLE table=test-table [id=3f6f1f0c7e8f4a35bdda92f8b763e00a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:05:17.901149 9876 log.cc:1196] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/ffafda5910c84eabb9699a445de16745
I20260928 02:05:17.902312 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8193071af6644bf69dc08024cbd6c37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:17.902698 9756 tablet_service.cc:1467] Processing CreateTablet for tablet 8193071af6644bf69dc08024cbd6c37f (DEFAULT_TABLE table=test-table [id=3f6f1f0c7e8f4a35bdda92f8b763e00a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:05:17.902925 9876 ts_tablet_manager.cc:1939] T ffafda5910c84eabb9699a445de16745 P 417e8867b6b74f14a2402ef52dd71ad6: Deleting consensus metadata
I20260928 02:05:17.903141 9756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8193071af6644bf69dc08024cbd6c37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:17.905329 9325 catalog_manager.cc:4841] TS 417e8867b6b74f14a2402ef52dd71ad6 (127.1.182.131:33033): tablet ffafda5910c84eabb9699a445de16745 (table test-table [id=a44e04f77b0344d1ad66e62aca77b304]) successfully deleted
I20260928 02:05:17.909374 9851 tablet_bootstrap.cc:492] T 8193071af6644bf69dc08024cbd6c37f P 9cc0277329224d2fb3a3aee237f4b6b5: Bootstrap starting.
I20260928 02:05:17.911049 9851 tablet_bootstrap.cc:654] T 8193071af6644bf69dc08024cbd6c37f P 9cc0277329224d2fb3a3aee237f4b6b5: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:17.911960 9477 tablet_service.cc:1467] Processing CreateTablet for tablet 8193071af6644bf69dc08024cbd6c37f (DEFAULT_TABLE table=test-table [id=3f6f1f0c7e8f4a35bdda92f8b763e00a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:05:17.912705 9477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8193071af6644bf69dc08024cbd6c37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:17.920121 9877 tablet_replica.cc:330] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8: stopping tablet replica
I20260928 02:05:17.920715 9851 tablet_bootstrap.cc:492] T 8193071af6644bf69dc08024cbd6c37f P 9cc0277329224d2fb3a3aee237f4b6b5: No bootstrap required, opened a new log
I20260928 02:05:17.920758 9877 raft_consensus.cc:2229] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:05:17.921003 9851 ts_tablet_manager.cc:1397] T 8193071af6644bf69dc08024cbd6c37f P 9cc0277329224d2fb3a3aee237f4b6b5: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.008s
I20260928 02:05:17.921576 9877 raft_consensus.cc:2258] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:05:17.921814 9852 tablet_bootstrap.cc:492] T 8193071af6644bf69dc08024cbd6c37f P 417e8867b6b74f14a2402ef52dd71ad6: Bootstrap starting.
I20260928 02:05:17.921984 9851 raft_consensus.cc:348] T 8193071af6644bf69dc08024cbd6c37f P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:17.922204 9851 raft_consensus.cc:374] T 8193071af6644bf69dc08024cbd6c37f P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:17.922293 9851 raft_consensus.cc:729] T 8193071af6644bf69dc08024cbd6c37f P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc0277329224d2fb3a3aee237f4b6b5, State: Initialized, Role: FOLLOWER
I20260928 02:05:17.922518 9851 consensus_queue.cc:260] T 8193071af6644bf69dc08024cbd6c37f P 9cc0277329224d2fb3a3aee237f4b6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:17.923054 9851 ts_tablet_manager.cc:1428] T 8193071af6644bf69dc08024cbd6c37f P 9cc0277329224d2fb3a3aee237f4b6b5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 02:05:17.923506 9852 tablet_bootstrap.cc:654] T 8193071af6644bf69dc08024cbd6c37f P 417e8867b6b74f14a2402ef52dd71ad6: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:17.924149 9853 tablet_bootstrap.cc:492] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8: Bootstrap starting.
I20260928 02:05:17.925510 9853 tablet_bootstrap.cc:654] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:17.926954 9877 ts_tablet_manager.cc:1905] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:17.928063 9852 tablet_bootstrap.cc:492] T 8193071af6644bf69dc08024cbd6c37f P 417e8867b6b74f14a2402ef52dd71ad6: No bootstrap required, opened a new log
I20260928 02:05:17.928210 9852 ts_tablet_manager.cc:1397] T 8193071af6644bf69dc08024cbd6c37f P 417e8867b6b74f14a2402ef52dd71ad6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260928 02:05:17.928982 9852 raft_consensus.cc:348] T 8193071af6644bf69dc08024cbd6c37f P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:17.929364 9852 raft_consensus.cc:374] T 8193071af6644bf69dc08024cbd6c37f P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:17.929474 9852 raft_consensus.cc:729] T 8193071af6644bf69dc08024cbd6c37f P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417e8867b6b74f14a2402ef52dd71ad6, State: Initialized, Role: FOLLOWER
I20260928 02:05:17.929726 9852 consensus_queue.cc:260] T 8193071af6644bf69dc08024cbd6c37f P 417e8867b6b74f14a2402ef52dd71ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:17.930320 9852 ts_tablet_manager.cc:1428] T 8193071af6644bf69dc08024cbd6c37f P 417e8867b6b74f14a2402ef52dd71ad6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 02:05:17.927484 9853 tablet_bootstrap.cc:492] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8: No bootstrap required, opened a new log
I20260928 02:05:17.930667 9853 ts_tablet_manager.cc:1397] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20260928 02:05:17.931759 9853 raft_consensus.cc:348] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:17.932073 9853 raft_consensus.cc:374] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:17.932235 9853 raft_consensus.cc:729] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecd8b31074bd483e889ac06dee87dfe8, State: Initialized, Role: FOLLOWER
I20260928 02:05:17.932660 9853 consensus_queue.cc:260] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:17.933388 9853 ts_tablet_manager.cc:1428] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 02:05:17.934093 9875 tablet_replica.cc:330] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5: stopping tablet replica
I20260928 02:05:17.934535 9875 raft_consensus.cc:2229] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:05:17.934993 9875 raft_consensus.cc:2258] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:05:17.937412 9875 ts_tablet_manager.cc:1905] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:17.938719 9877 ts_tablet_manager.cc:1918] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:05:17.938879 9877 log.cc:1196] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-0/wal/wals/ffafda5910c84eabb9699a445de16745
I20260928 02:05:17.939450 9877 ts_tablet_manager.cc:1939] T ffafda5910c84eabb9699a445de16745 P ecd8b31074bd483e889ac06dee87dfe8: Deleting consensus metadata
I20260928 02:05:17.941087 9325 catalog_manager.cc:4841] TS ecd8b31074bd483e889ac06dee87dfe8 (127.1.182.129:35749): tablet ffafda5910c84eabb9699a445de16745 (table test-table [id=a44e04f77b0344d1ad66e62aca77b304]) successfully deleted
I20260928 02:05:17.942694 9875 ts_tablet_manager.cc:1918] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:05:17.942852 9875 log.cc:1196] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/ffafda5910c84eabb9699a445de16745
I20260928 02:05:17.943414 9875 ts_tablet_manager.cc:1939] T ffafda5910c84eabb9699a445de16745 P 9cc0277329224d2fb3a3aee237f4b6b5: Deleting consensus metadata
I20260928 02:05:17.944859 9325 catalog_manager.cc:4841] TS 9cc0277329224d2fb3a3aee237f4b6b5 (127.1.182.130:43485): tablet ffafda5910c84eabb9699a445de16745 (table test-table [id=a44e04f77b0344d1ad66e62aca77b304]) successfully deleted
I20260928 02:05:18.134087 9862 raft_consensus.cc:482] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:18.134433 9862 raft_consensus.cc:504] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:18.135190 9862 leader_election.cc:288] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cc0277329224d2fb3a3aee237f4b6b5 (127.1.182.130:43485), 417e8867b6b74f14a2402ef52dd71ad6 (127.1.182.131:33033)
I20260928 02:05:18.135797 9642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8193071af6644bf69dc08024cbd6c37f" candidate_uuid: "ecd8b31074bd483e889ac06dee87dfe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" is_pre_election: true
I20260928 02:05:18.135957 9781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8193071af6644bf69dc08024cbd6c37f" candidate_uuid: "ecd8b31074bd483e889ac06dee87dfe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "417e8867b6b74f14a2402ef52dd71ad6" is_pre_election: true
I20260928 02:05:18.136049 9642 raft_consensus.cc:2454] T 8193071af6644bf69dc08024cbd6c37f P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecd8b31074bd483e889ac06dee87dfe8 in term 0.
I20260928 02:05:18.136215 9781 raft_consensus.cc:2454] T 8193071af6644bf69dc08024cbd6c37f P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecd8b31074bd483e889ac06dee87dfe8 in term 0.
I20260928 02:05:18.136511 9428 leader_election.cc:302] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cc0277329224d2fb3a3aee237f4b6b5, ecd8b31074bd483e889ac06dee87dfe8; no voters:
I20260928 02:05:18.136775 9862 raft_consensus.cc:2781] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:05:18.136895 9862 raft_consensus.cc:482] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:05:18.137037 9862 raft_consensus.cc:3037] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:18.139254 9862 raft_consensus.cc:504] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:18.140161 9862 leader_election.cc:288] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 [CANDIDATE]: Term 1 election: Requested vote from peers 9cc0277329224d2fb3a3aee237f4b6b5 (127.1.182.130:43485), 417e8867b6b74f14a2402ef52dd71ad6 (127.1.182.131:33033)
I20260928 02:05:18.140507 9642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8193071af6644bf69dc08024cbd6c37f" candidate_uuid: "ecd8b31074bd483e889ac06dee87dfe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc0277329224d2fb3a3aee237f4b6b5"
I20260928 02:05:18.140717 9642 raft_consensus.cc:3037] T 8193071af6644bf69dc08024cbd6c37f P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:18.140710 9781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8193071af6644bf69dc08024cbd6c37f" candidate_uuid: "ecd8b31074bd483e889ac06dee87dfe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "417e8867b6b74f14a2402ef52dd71ad6"
I20260928 02:05:18.140976 9781 raft_consensus.cc:3037] T 8193071af6644bf69dc08024cbd6c37f P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:18.143404 9781 raft_consensus.cc:2454] T 8193071af6644bf69dc08024cbd6c37f P 417e8867b6b74f14a2402ef52dd71ad6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecd8b31074bd483e889ac06dee87dfe8 in term 1.
I20260928 02:05:18.143694 9642 raft_consensus.cc:2454] T 8193071af6644bf69dc08024cbd6c37f P 9cc0277329224d2fb3a3aee237f4b6b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecd8b31074bd483e889ac06dee87dfe8 in term 1.
I20260928 02:05:18.143847 9428 leader_election.cc:302] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 417e8867b6b74f14a2402ef52dd71ad6, ecd8b31074bd483e889ac06dee87dfe8; no voters:
I20260928 02:05:18.144243 9862 raft_consensus.cc:2781] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:05:18.144413 9862 raft_consensus.cc:686] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 [term 1 LEADER]: Becoming Leader. State: Replica: ecd8b31074bd483e889ac06dee87dfe8, State: Running, Role: LEADER
I20260928 02:05:18.144697 9862 consensus_queue.cc:237] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:18.147711 9341 catalog_manager.cc:5495] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecd8b31074bd483e889ac06dee87dfe8 (127.1.182.129). New cstate: current_term: 1 leader_uuid: "ecd8b31074bd483e889ac06dee87dfe8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:05:18 dist-test-slave-23bp krb5kdc[9296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561114, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.190@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM
I20260928 02:05:18.215349 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:18.181624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:36451 (local address 127.0.0.1:47752)
0928 02:05:18.197626 (+ 16002us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:18.197653 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:18.197821 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:18.198247 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:18.198255 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:18.199215 (+ 960us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:18.199545 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:18.199557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.201065 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.201071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:18.202646 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:18.202659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.203676 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.203684 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:18.203880 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:18.204732 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:18.204757 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:18.207809 (+ 3052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:18.211978 (+ 4169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:18.211992 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:18.212001 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:18.212582 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:18.213042 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:18.213049 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:18.213055 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:18.213245 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:18.213916 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:18.213926 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:18.214161 (+ 235us) client_negotiation.cc:769] Sending connection context
0928 02:05:18.214346 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 02:05:18.214817 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15770,"mutex_wait_us":23,"thread_start_us":15670,"threads_started":1}
I20260928 02:05:18.215466 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:18.181506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47752 (local address 127.1.182.190:36451)
0928 02:05:18.197152 (+ 15646us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:18.197159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:18.197776 (+ 617us) server_negotiation.cc:407] Connection header received
0928 02:05:18.197965 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:18.197971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:18.198062 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:18.198205 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:18.199733 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.200897 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.202795 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.203468 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.203585 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:18.208072 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:18.208079 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:18.208084 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:18.208120 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:18.211824 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:18.212758 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:18.212766 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:18.212770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:18.212916 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:18.213446 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:18.213451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:18.213455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:18.213794 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:18.213946 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:18.214472 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 02:05:18.214970 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":244,"server-negotiator.queue_time_us":15490,"thread_start_us":15999,"threads_started":1}
I20260928 02:05:18.221315 9341 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:47752:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:05:18.222498 9341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:05:18.230017 9756 tablet_service.cc:1467] Processing CreateTablet for tablet 4b9c57931bcd48a1b13a8be705d46de4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b0598a30aad4fa28b8aaba93633a5cc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:05:18.230158 9477 tablet_service.cc:1467] Processing CreateTablet for tablet 4b9c57931bcd48a1b13a8be705d46de4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b0598a30aad4fa28b8aaba93633a5cc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:05:18.230618 9756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b9c57931bcd48a1b13a8be705d46de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:18.230638 9477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b9c57931bcd48a1b13a8be705d46de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:18.235668 9617 tablet_service.cc:1467] Processing CreateTablet for tablet 4b9c57931bcd48a1b13a8be705d46de4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b0598a30aad4fa28b8aaba93633a5cc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:05:18.236280 9617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b9c57931bcd48a1b13a8be705d46de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:18.237979 9852 tablet_bootstrap.cc:492] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6: Bootstrap starting.
I20260928 02:05:18.239001 9853 tablet_bootstrap.cc:492] T 4b9c57931bcd48a1b13a8be705d46de4 P ecd8b31074bd483e889ac06dee87dfe8: Bootstrap starting.
I20260928 02:05:18.239660 9852 tablet_bootstrap.cc:654] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:18.240980 9853 tablet_bootstrap.cc:654] T 4b9c57931bcd48a1b13a8be705d46de4 P ecd8b31074bd483e889ac06dee87dfe8: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:18.245424 9851 tablet_bootstrap.cc:492] T 4b9c57931bcd48a1b13a8be705d46de4 P 9cc0277329224d2fb3a3aee237f4b6b5: Bootstrap starting.
I20260928 02:05:18.247707 9851 tablet_bootstrap.cc:654] T 4b9c57931bcd48a1b13a8be705d46de4 P 9cc0277329224d2fb3a3aee237f4b6b5: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:18.251312 9851 tablet_bootstrap.cc:492] T 4b9c57931bcd48a1b13a8be705d46de4 P 9cc0277329224d2fb3a3aee237f4b6b5: No bootstrap required, opened a new log
I20260928 02:05:18.251488 9851 ts_tablet_manager.cc:1397] T 4b9c57931bcd48a1b13a8be705d46de4 P 9cc0277329224d2fb3a3aee237f4b6b5: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260928 02:05:18.252359 9852 tablet_bootstrap.cc:492] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6: No bootstrap required, opened a new log
I20260928 02:05:18.252220 9851 raft_consensus.cc:348] T 4b9c57931bcd48a1b13a8be705d46de4 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } }
I20260928 02:05:18.252465 9851 raft_consensus.cc:374] T 4b9c57931bcd48a1b13a8be705d46de4 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:18.252522 9852 ts_tablet_manager.cc:1397] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260928 02:05:18.252547 9851 raft_consensus.cc:729] T 4b9c57931bcd48a1b13a8be705d46de4 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc0277329224d2fb3a3aee237f4b6b5, State: Initialized, Role: FOLLOWER
I20260928 02:05:18.252799 9851 consensus_queue.cc:260] T 4b9c57931bcd48a1b13a8be705d46de4 P 9cc0277329224d2fb3a3aee237f4b6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } }
I20260928 02:05:18.253324 9851 ts_tablet_manager.cc:1428] T 4b9c57931bcd48a1b13a8be705d46de4 P 9cc0277329224d2fb3a3aee237f4b6b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:05:18.253288 9852 raft_consensus.cc:348] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } }
I20260928 02:05:18.253569 9852 raft_consensus.cc:374] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:18.253682 9852 raft_consensus.cc:729] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417e8867b6b74f14a2402ef52dd71ad6, State: Initialized, Role: FOLLOWER
I20260928 02:05:18.253983 9852 consensus_queue.cc:260] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } }
I20260928 02:05:18.254424 9857 tablet_replica.cc:439] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } }
I20260928 02:05:18.254770 9857 tablet_replica.cc:442] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:05:18.254897 9858 tablet_replica.cc:439] T 4b9c57931bcd48a1b13a8be705d46de4 P 9cc0277329224d2fb3a3aee237f4b6b5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } }
I20260928 02:05:18.255277 9858 tablet_replica.cc:442] T 4b9c57931bcd48a1b13a8be705d46de4 P 9cc0277329224d2fb3a3aee237f4b6b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:05:18.256390 9853 tablet_bootstrap.cc:492] T 4b9c57931bcd48a1b13a8be705d46de4 P ecd8b31074bd483e889ac06dee87dfe8: No bootstrap required, opened a new log
I20260928 02:05:18.256625 9853 ts_tablet_manager.cc:1397] T 4b9c57931bcd48a1b13a8be705d46de4 P ecd8b31074bd483e889ac06dee87dfe8: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.004s
I20260928 02:05:18.257509 9853 raft_consensus.cc:348] T 4b9c57931bcd48a1b13a8be705d46de4 P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } }
I20260928 02:05:18.257822 9853 raft_consensus.cc:374] T 4b9c57931bcd48a1b13a8be705d46de4 P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:18.257959 9853 raft_consensus.cc:729] T 4b9c57931bcd48a1b13a8be705d46de4 P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecd8b31074bd483e889ac06dee87dfe8, State: Initialized, Role: FOLLOWER
I20260928 02:05:18.257385 9852 ts_tablet_manager.cc:1428] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260928 02:05:18.258267 9853 consensus_queue.cc:260] T 4b9c57931bcd48a1b13a8be705d46de4 P ecd8b31074bd483e889ac06dee87dfe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } }
I20260928 02:05:18.258872 9853 ts_tablet_manager.cc:1428] T 4b9c57931bcd48a1b13a8be705d46de4 P ecd8b31074bd483e889ac06dee87dfe8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:05:18.258773 9862 tablet_replica.cc:439] T 4b9c57931bcd48a1b13a8be705d46de4 P ecd8b31074bd483e889ac06dee87dfe8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } }
I20260928 02:05:18.260037 9862 tablet_replica.cc:442] T 4b9c57931bcd48a1b13a8be705d46de4 P ecd8b31074bd483e889ac06dee87dfe8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:05:18.329282 9857 raft_consensus.cc:482] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:18.329553 9857 raft_consensus.cc:504] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } }
I20260928 02:05:18.331110 9857 leader_election.cc:288] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecd8b31074bd483e889ac06dee87dfe8 (127.1.182.129:35749), 9cc0277329224d2fb3a3aee237f4b6b5 (127.1.182.130:43485)
I20260928 02:05:18.339402 9850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:18.331403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:59569 (local address 127.1.182.129:35749)
0928 02:05:18.331689 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:18.331698 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:18.332190 (+ 492us) server_negotiation.cc:407] Connection header received
0928 02:05:18.332430 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:18.332446 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:18.332547 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:18.332609 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:18.333394 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.334509 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.337078 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.338245 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.338362 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:18.338434 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:18.338941 (+ 507us) server_negotiation.cc:299] Negotiation successful
0928 02:05:18.339094 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:05:18.340256 9892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:18.331323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:35749 (local address 127.1.182.131:59569)
0928 02:05:18.332091 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:18.332110 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:18.332289 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:18.332837 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:18.332841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:18.332869 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:18.333213 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:18.333224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.334753 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.334759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:18.336904 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:18.336918 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.338573 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.338578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:18.338766 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:18.338799 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:05:18.339827 (+ 1028us) client_negotiation.cc:240] Negotiation successful
0928 02:05:18.339961 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":444,"threads_started":1}
I20260928 02:05:18.342123 9503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9c57931bcd48a1b13a8be705d46de4" candidate_uuid: "417e8867b6b74f14a2402ef52dd71ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecd8b31074bd483e889ac06dee87dfe8" is_pre_election: true
I20260928 02:05:18.342542 9503 raft_consensus.cc:2454] T 4b9c57931bcd48a1b13a8be705d46de4 P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 417e8867b6b74f14a2402ef52dd71ad6 in term 0.
I20260928 02:05:18.343127 9707 leader_election.cc:302] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 417e8867b6b74f14a2402ef52dd71ad6, ecd8b31074bd483e889ac06dee87dfe8; no voters:
I20260928 02:05:18.343525 9857 raft_consensus.cc:2781] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:05:18.343654 9857 raft_consensus.cc:482] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:05:18.343765 9857 raft_consensus.cc:3037] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:18.346354 9857 raft_consensus.cc:504] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } }
I20260928 02:05:18.347000 9857 leader_election.cc:288] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6 [CANDIDATE]: Term 1 election: Requested vote from peers ecd8b31074bd483e889ac06dee87dfe8 (127.1.182.129:35749), 9cc0277329224d2fb3a3aee237f4b6b5 (127.1.182.130:43485)
I20260928 02:05:18.347649 9503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9c57931bcd48a1b13a8be705d46de4" candidate_uuid: "417e8867b6b74f14a2402ef52dd71ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecd8b31074bd483e889ac06dee87dfe8"
I20260928 02:05:18.347927 9892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:18.332186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:43485 (local address 127.1.182.131:53831)
0928 02:05:18.340633 (+ 8447us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:18.340652 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:18.340747 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:18.341533 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:18.341538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:18.341565 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:18.341733 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:18.341742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.343061 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.343107 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:18.345130 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:18.345144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.347251 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.347257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:18.347443 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:18.347478 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:05:18.347542 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 02:05:18.347670 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8359,"thread_start_us":9058,"threads_started":1}
I20260928 02:05:18.347849 9503 raft_consensus.cc:3037] T 4b9c57931bcd48a1b13a8be705d46de4 P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:18.348730 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:18.332288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:53831 (local address 127.1.182.130:43485)
0928 02:05:18.341247 (+ 8959us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:18.341253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:18.341270 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:05:18.341331 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:18.341336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:18.341397 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:18.341506 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:18.341912 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.342910 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.345298 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.346492 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.348191 (+ 1699us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:18.348258 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:18.348306 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 02:05:18.348434 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8839,"thread_start_us":8671,"threads_started":1}
I20260928 02:05:18.350382 9503 raft_consensus.cc:2454] T 4b9c57931bcd48a1b13a8be705d46de4 P ecd8b31074bd483e889ac06dee87dfe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 417e8867b6b74f14a2402ef52dd71ad6 in term 1.
I20260928 02:05:18.350816 9707 leader_election.cc:302] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 417e8867b6b74f14a2402ef52dd71ad6, ecd8b31074bd483e889ac06dee87dfe8; no voters:
I20260928 02:05:18.351097 9857 raft_consensus.cc:2781] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:05:18.351094 9642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9c57931bcd48a1b13a8be705d46de4" candidate_uuid: "417e8867b6b74f14a2402ef52dd71ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" is_pre_election: true
I20260928 02:05:18.351200 9641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9c57931bcd48a1b13a8be705d46de4" candidate_uuid: "417e8867b6b74f14a2402ef52dd71ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc0277329224d2fb3a3aee237f4b6b5"
I20260928 02:05:18.351387 9642 raft_consensus.cc:2454] T 4b9c57931bcd48a1b13a8be705d46de4 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 417e8867b6b74f14a2402ef52dd71ad6 in term 0.
I20260928 02:05:18.351928 9857 raft_consensus.cc:686] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6 [term 1 LEADER]: Becoming Leader. State: Replica: 417e8867b6b74f14a2402ef52dd71ad6, State: Running, Role: LEADER
I20260928 02:05:18.352342 9857 consensus_queue.cc:237] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } }
I20260928 02:05:18.354442 9895 tablet_replica.cc:439] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6: TxnStatusTablet state changed. Reason: New leader 417e8867b6b74f14a2402ef52dd71ad6. Latest consensus state: current_term: 1 leader_uuid: "417e8867b6b74f14a2402ef52dd71ad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } }
I20260928 02:05:18.354771 9895 tablet_replica.cc:442] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:05:18.355631 9897 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:05:18.355832 9897 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:05:18.356283 9341 catalog_manager.cc:5495] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 417e8867b6b74f14a2402ef52dd71ad6 (127.1.182.131). New cstate: current_term: 1 leader_uuid: "417e8867b6b74f14a2402ef52dd71ad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } health_report { overall_health: UNKNOWN } } }
I20260928 02:05:18.385214 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:18.379403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:33033 (local address 127.0.0.1:42416)
0928 02:05:18.379600 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:18.379621 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:18.379771 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:18.380455 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:18.380460 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:18.380472 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:05:18.380672 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:18.380685 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.381739 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.381746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:18.382705 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:18.382716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.383486 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.383492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:18.383652 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:18.383705 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:05:18.384596 (+ 891us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:05:18.384615 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:05:18.384689 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:05:18.384843 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 02:05:18.385215 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:18.379303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42416 (local address 127.1.182.131:33033)
0928 02:05:18.380056 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:18.380060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:18.380076 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:05:18.380153 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:18.380156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:18.380250 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:18.380438 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:05:18.380819 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.381610 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.382844 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.383341 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.383475 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:18.383955 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:05:18.384490 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:05:18.384650 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:18.384733 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 02:05:18.384832 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":418,"threads_started":1}
I20260928 02:05:18.389591 9641 raft_consensus.cc:3037] T 4b9c57931bcd48a1b13a8be705d46de4 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:18.389923 9503 raft_consensus.cc:1261] T 4b9c57931bcd48a1b13a8be705d46de4 P ecd8b31074bd483e889ac06dee87dfe8 [term 1 FOLLOWER]: Refusing update from remote peer 417e8867b6b74f14a2402ef52dd71ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:05:18.390717 9895 consensus_queue.cc:1035] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:05:18.392521 9641 raft_consensus.cc:1261] T 4b9c57931bcd48a1b13a8be705d46de4 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 1 FOLLOWER]: Refusing update from remote peer 417e8867b6b74f14a2402ef52dd71ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:05:18.393005 9895 consensus_queue.cc:1035] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:05:18.399891 9858 tablet_replica.cc:439] T 4b9c57931bcd48a1b13a8be705d46de4 P 9cc0277329224d2fb3a3aee237f4b6b5: TxnStatusTablet state changed. Reason: New leader 417e8867b6b74f14a2402ef52dd71ad6. Latest consensus state: current_term: 1 leader_uuid: "417e8867b6b74f14a2402ef52dd71ad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } }
I20260928 02:05:18.400199 9858 tablet_replica.cc:442] T 4b9c57931bcd48a1b13a8be705d46de4 P 9cc0277329224d2fb3a3aee237f4b6b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:05:18.402352 9901 mvcc.cc:205] Tried to move back new op lower bound from 7334138340917407744 to 7334138340774612992. Current Snapshot: MvccSnapshot[applied={T|T < 7334138340917407744}]
I20260928 02:05:18.404564 9858 tablet_replica.cc:439] T 4b9c57931bcd48a1b13a8be705d46de4 P 9cc0277329224d2fb3a3aee237f4b6b5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "417e8867b6b74f14a2402ef52dd71ad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } }
I20260928 02:05:18.404856 9858 tablet_replica.cc:442] T 4b9c57931bcd48a1b13a8be705d46de4 P 9cc0277329224d2fb3a3aee237f4b6b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:05:18.409543 9857 tablet_replica.cc:439] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "417e8867b6b74f14a2402ef52dd71ad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } }
I20260928 02:05:18.409905 9857 tablet_replica.cc:442] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:05:18.415664 9862 tablet_replica.cc:439] T 4b9c57931bcd48a1b13a8be705d46de4 P ecd8b31074bd483e889ac06dee87dfe8: TxnStatusTablet state changed. Reason: New leader 417e8867b6b74f14a2402ef52dd71ad6. Latest consensus state: current_term: 1 leader_uuid: "417e8867b6b74f14a2402ef52dd71ad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } }
I20260928 02:05:18.415938 9862 tablet_replica.cc:442] T 4b9c57931bcd48a1b13a8be705d46de4 P ecd8b31074bd483e889ac06dee87dfe8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:05:18.420214 9857 tablet_replica.cc:439] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "417e8867b6b74f14a2402ef52dd71ad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } }
I20260928 02:05:18.420514 9857 tablet_replica.cc:442] T 4b9c57931bcd48a1b13a8be705d46de4 P 417e8867b6b74f14a2402ef52dd71ad6: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:05:18.423983 9862 tablet_replica.cc:439] T 4b9c57931bcd48a1b13a8be705d46de4 P ecd8b31074bd483e889ac06dee87dfe8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "417e8867b6b74f14a2402ef52dd71ad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } }
I20260928 02:05:18.424244 9862 tablet_replica.cc:442] T 4b9c57931bcd48a1b13a8be705d46de4 P ecd8b31074bd483e889ac06dee87dfe8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:05:18.437922 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:18.432556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:55573 (local address 127.1.182.131:33033)
0928 02:05:18.432744 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:18.432752 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:18.433130 (+ 378us) server_negotiation.cc:407] Connection header received
0928 02:05:18.433360 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:18.433364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:18.433425 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:18.433554 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:05:18.434002 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.434776 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.436200 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.436617 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.436770 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:18.437211 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:05:18.437307 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:05:18.437407 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:18.437562 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 02:05:18.437635 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:05:18.438087 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:18.432460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:33033 (local address 127.1.182.129:55573)
0928 02:05:18.433023 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:18.433038 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:18.433202 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:18.433569 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:18.433574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:18.433587 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:05:18.433860 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:18.433868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.434916 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.434922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:18.436057 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:18.436068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.436825 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.436832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:18.436979 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:18.437028 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:05:18.437406 (+ 378us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:05:18.437419 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:05:18.437542 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 02:05:18.437667 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":292,"threads_started":1}
I20260928 02:05:18.446820 9781 raft_consensus.cc:1261] T 8193071af6644bf69dc08024cbd6c37f P 417e8867b6b74f14a2402ef52dd71ad6 [term 1 FOLLOWER]: Refusing update from remote peer ecd8b31074bd483e889ac06dee87dfe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:05:18.446895 9641 raft_consensus.cc:1261] T 8193071af6644bf69dc08024cbd6c37f P 9cc0277329224d2fb3a3aee237f4b6b5 [term 1 FOLLOWER]: Refusing update from remote peer ecd8b31074bd483e889ac06dee87dfe8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:05:18.447331 9872 consensus_queue.cc:1035] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:05:18.447690 9862 consensus_queue.cc:1035] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:05:18.482596 9850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:18.475796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:35913 (local address 127.1.182.129:35749)
0928 02:05:18.475983 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:18.475991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:18.476791 (+ 800us) server_negotiation.cc:407] Connection header received
0928 02:05:18.477081 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:18.477088 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:18.477166 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:18.477263 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:05:18.477893 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.478956 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.480503 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.481016 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.481157 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:18.481857 (+ 700us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:05:18.481999 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:05:18.482099 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:18.482206 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 02:05:18.482290 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:05:18.482628 9916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:18.475921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:35749 (local address 127.1.182.131:35913)
0928 02:05:18.476657 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:18.476689 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:18.476856 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:18.477292 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:18.477297 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:18.477310 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:05:18.477692 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:18.477705 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.479097 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.479105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:18.480338 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:18.480349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.481209 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.481217 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:18.481361 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:18.481441 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:05:18.482097 (+ 656us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:05:18.482107 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:05:18.482188 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:05:18.482319 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":324,"threads_started":1}
I20260928 02:05:18.518306 9340 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:05:18.518591 9340 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:05:18.521943 9340 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43: Sending DeleteTablet for 3 replicas of tablet 8193071af6644bf69dc08024cbd6c37f
I20260928 02:05:18.522671 9477 tablet_service.cc:1514] Processing DeleteTablet for tablet 8193071af6644bf69dc08024cbd6c37f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:05:18 UTC) from {username='oryx', principal='oryx/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:50460
I20260928 02:05:18.522858 9617 tablet_service.cc:1514] Processing DeleteTablet for tablet 8193071af6644bf69dc08024cbd6c37f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:05:18 UTC) from {username='oryx', principal='oryx/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:50000
I20260928 02:05:18.523356 9919 tablet_replica.cc:330] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8: stopping tablet replica
I20260928 02:05:18.523484 9756 tablet_service.cc:1514] Processing DeleteTablet for tablet 8193071af6644bf69dc08024cbd6c37f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:05:18 UTC) from {username='oryx', principal='oryx/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:49008
I20260928 02:05:18.523552 9919 raft_consensus.cc:2229] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:05:18.523897 9919 raft_consensus.cc:2258] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:05:18.525094 9919 ts_tablet_manager.cc:1905] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:18.532378 9919 ts_tablet_manager.cc:1918] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:05:18.532531 9919 log.cc:1196] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-0/wal/wals/8193071af6644bf69dc08024cbd6c37f
I20260928 02:05:18.532990 9919 ts_tablet_manager.cc:1939] T 8193071af6644bf69dc08024cbd6c37f P ecd8b31074bd483e889ac06dee87dfe8: Deleting consensus metadata
I20260928 02:05:18.534065 9325 catalog_manager.cc:4841] TS ecd8b31074bd483e889ac06dee87dfe8 (127.1.182.129:35749): tablet 8193071af6644bf69dc08024cbd6c37f (table test-table [id=3f6f1f0c7e8f4a35bdda92f8b763e00a]) successfully deleted
I20260928 02:05:18.536535 9921 tablet_replica.cc:330] T 8193071af6644bf69dc08024cbd6c37f P 417e8867b6b74f14a2402ef52dd71ad6: stopping tablet replica
I20260928 02:05:18.536875 9921 raft_consensus.cc:2229] T 8193071af6644bf69dc08024cbd6c37f P 417e8867b6b74f14a2402ef52dd71ad6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:05:18.537256 9921 raft_consensus.cc:2258] T 8193071af6644bf69dc08024cbd6c37f P 417e8867b6b74f14a2402ef52dd71ad6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:05:18.539209 9921 ts_tablet_manager.cc:1905] T 8193071af6644bf69dc08024cbd6c37f P 417e8867b6b74f14a2402ef52dd71ad6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:18.542254 9920 tablet_replica.cc:330] T 8193071af6644bf69dc08024cbd6c37f P 9cc0277329224d2fb3a3aee237f4b6b5: stopping tablet replica
I20260928 02:05:18.542475 9920 raft_consensus.cc:2229] T 8193071af6644bf69dc08024cbd6c37f P 9cc0277329224d2fb3a3aee237f4b6b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:05:18.542753 9920 raft_consensus.cc:2258] T 8193071af6644bf69dc08024cbd6c37f P 9cc0277329224d2fb3a3aee237f4b6b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:05:18.543936 9920 ts_tablet_manager.cc:1905] T 8193071af6644bf69dc08024cbd6c37f P 9cc0277329224d2fb3a3aee237f4b6b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:18.548374 9921 ts_tablet_manager.cc:1918] T 8193071af6644bf69dc08024cbd6c37f P 417e8867b6b74f14a2402ef52dd71ad6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:05:18.548606 9921 log.cc:1196] T 8193071af6644bf69dc08024cbd6c37f P 417e8867b6b74f14a2402ef52dd71ad6: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/8193071af6644bf69dc08024cbd6c37f
I20260928 02:05:18.549216 9921 ts_tablet_manager.cc:1939] T 8193071af6644bf69dc08024cbd6c37f P 417e8867b6b74f14a2402ef52dd71ad6: Deleting consensus metadata
I20260928 02:05:18.550532 9325 catalog_manager.cc:4841] TS 417e8867b6b74f14a2402ef52dd71ad6 (127.1.182.131:33033): tablet 8193071af6644bf69dc08024cbd6c37f (table test-table [id=3f6f1f0c7e8f4a35bdda92f8b763e00a]) successfully deleted
I20260928 02:05:18.551741 9920 ts_tablet_manager.cc:1918] T 8193071af6644bf69dc08024cbd6c37f P 9cc0277329224d2fb3a3aee237f4b6b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:05:18.551895 9920 log.cc:1196] T 8193071af6644bf69dc08024cbd6c37f P 9cc0277329224d2fb3a3aee237f4b6b5: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/8193071af6644bf69dc08024cbd6c37f
I20260928 02:05:18.552384 9920 ts_tablet_manager.cc:1939] T 8193071af6644bf69dc08024cbd6c37f P 9cc0277329224d2fb3a3aee237f4b6b5: Deleting consensus metadata
I20260928 02:05:18.553618 9325 catalog_manager.cc:4841] TS 9cc0277329224d2fb3a3aee237f4b6b5 (127.1.182.130:43485): tablet 8193071af6644bf69dc08024cbd6c37f (table test-table [id=3f6f1f0c7e8f4a35bdda92f8b763e00a]) successfully deleted
Sep 28 02:05:18 dist-test-slave-23bp krb5kdc[9296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:05:18 dist-test-slave-23bp krb5kdc[9296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:18 dist-test-slave-23bp krb5kdc[9296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:05:18.596851 9887 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:18.584523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53734 (local address 127.1.182.190:36451)
0928 02:05:18.584743 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:18.584752 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:18.585263 (+ 511us) server_negotiation.cc:407] Connection header received
0928 02:05:18.585523 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:18.585528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:18.585606 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:18.585692 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:18.587324 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.588042 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.589904 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.590329 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.590421 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:18.596618 (+ 6197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53734: BlockingRecv error: recv got EOF from 127.0.0.1:53734 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 02:05:18 dist-test-slave-23bp krb5kdc[9296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM
I20260928 02:05:18.615038 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:18.599908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53748 (local address 127.1.182.190:36451)
0928 02:05:18.600077 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:18.600083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:18.600373 (+ 290us) server_negotiation.cc:407] Connection header received
0928 02:05:18.600512 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:18.600518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:18.600593 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:18.600707 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:18.601807 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.602964 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.604196 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.604846 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.604979 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:18.608781 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:18.608788 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:18.608794 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:18.608835 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:18.612281 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:18.612881 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:18.612888 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:18.612892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:18.613061 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:18.613531 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:18.613536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:18.613540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:18.613912 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:18.614166 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:18.614445 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 02:05:18.614681 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:05:18.618741 9340 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:05:18.619446 9340 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:05:18.625890 9477 tablet_service.cc:1467] Processing CreateTablet for tablet 7e5edea96d6441f18980b2e7fe8c71f0 (DEFAULT_TABLE table=test-table [id=db270d4a6d9c4cb49761e016d9c15667]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:05:18.626233 9617 tablet_service.cc:1467] Processing CreateTablet for tablet 7e5edea96d6441f18980b2e7fe8c71f0 (DEFAULT_TABLE table=test-table [id=db270d4a6d9c4cb49761e016d9c15667]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:05:18.626356 9477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5edea96d6441f18980b2e7fe8c71f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:18.626509 9756 tablet_service.cc:1467] Processing CreateTablet for tablet 7e5edea96d6441f18980b2e7fe8c71f0 (DEFAULT_TABLE table=test-table [id=db270d4a6d9c4cb49761e016d9c15667]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:05:18.626642 9617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5edea96d6441f18980b2e7fe8c71f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:18.626904 9756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5edea96d6441f18980b2e7fe8c71f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:18.632913 9853 tablet_bootstrap.cc:492] T 7e5edea96d6441f18980b2e7fe8c71f0 P ecd8b31074bd483e889ac06dee87dfe8: Bootstrap starting.
I20260928 02:05:18.634043 9852 tablet_bootstrap.cc:492] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6: Bootstrap starting.
I20260928 02:05:18.634503 9853 tablet_bootstrap.cc:654] T 7e5edea96d6441f18980b2e7fe8c71f0 P ecd8b31074bd483e889ac06dee87dfe8: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:18.635836 9852 tablet_bootstrap.cc:654] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:18.636910 9851 tablet_bootstrap.cc:492] T 7e5edea96d6441f18980b2e7fe8c71f0 P 9cc0277329224d2fb3a3aee237f4b6b5: Bootstrap starting.
I20260928 02:05:18.638434 9851 tablet_bootstrap.cc:654] T 7e5edea96d6441f18980b2e7fe8c71f0 P 9cc0277329224d2fb3a3aee237f4b6b5: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:18.644761 9853 tablet_bootstrap.cc:492] T 7e5edea96d6441f18980b2e7fe8c71f0 P ecd8b31074bd483e889ac06dee87dfe8: No bootstrap required, opened a new log
I20260928 02:05:18.644942 9853 ts_tablet_manager.cc:1397] T 7e5edea96d6441f18980b2e7fe8c71f0 P ecd8b31074bd483e889ac06dee87dfe8: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20260928 02:05:18.645929 9853 raft_consensus.cc:348] T 7e5edea96d6441f18980b2e7fe8c71f0 P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:18.646250 9853 raft_consensus.cc:374] T 7e5edea96d6441f18980b2e7fe8c71f0 P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:18.646359 9853 raft_consensus.cc:729] T 7e5edea96d6441f18980b2e7fe8c71f0 P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecd8b31074bd483e889ac06dee87dfe8, State: Initialized, Role: FOLLOWER
I20260928 02:05:18.646632 9853 consensus_queue.cc:260] T 7e5edea96d6441f18980b2e7fe8c71f0 P ecd8b31074bd483e889ac06dee87dfe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:18.647827 9852 tablet_bootstrap.cc:492] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6: No bootstrap required, opened a new log
I20260928 02:05:18.647989 9852 ts_tablet_manager.cc:1397] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260928 02:05:18.648730 9852 raft_consensus.cc:348] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:18.649045 9852 raft_consensus.cc:374] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:18.649163 9852 raft_consensus.cc:729] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417e8867b6b74f14a2402ef52dd71ad6, State: Initialized, Role: FOLLOWER
I20260928 02:05:18.649502 9853 ts_tablet_manager.cc:1428] T 7e5edea96d6441f18980b2e7fe8c71f0 P ecd8b31074bd483e889ac06dee87dfe8: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260928 02:05:18.649402 9852 consensus_queue.cc:260] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:18.651839 9852 ts_tablet_manager.cc:1428] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260928 02:05:18.652463 9851 tablet_bootstrap.cc:492] T 7e5edea96d6441f18980b2e7fe8c71f0 P 9cc0277329224d2fb3a3aee237f4b6b5: No bootstrap required, opened a new log
I20260928 02:05:18.652729 9851 ts_tablet_manager.cc:1397] T 7e5edea96d6441f18980b2e7fe8c71f0 P 9cc0277329224d2fb3a3aee237f4b6b5: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.003s
I20260928 02:05:18.653406 9851 raft_consensus.cc:348] T 7e5edea96d6441f18980b2e7fe8c71f0 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:18.653648 9851 raft_consensus.cc:374] T 7e5edea96d6441f18980b2e7fe8c71f0 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:18.653733 9851 raft_consensus.cc:729] T 7e5edea96d6441f18980b2e7fe8c71f0 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc0277329224d2fb3a3aee237f4b6b5, State: Initialized, Role: FOLLOWER
I20260928 02:05:18.653920 9851 consensus_queue.cc:260] T 7e5edea96d6441f18980b2e7fe8c71f0 P 9cc0277329224d2fb3a3aee237f4b6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:18.654839 9851 ts_tablet_manager.cc:1428] T 7e5edea96d6441f18980b2e7fe8c71f0 P 9cc0277329224d2fb3a3aee237f4b6b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 02:05:18.706743 9857 raft_consensus.cc:482] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:18.706964 9857 raft_consensus.cc:504] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:18.707628 9857 leader_election.cc:288] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecd8b31074bd483e889ac06dee87dfe8 (127.1.182.129:35749), 9cc0277329224d2fb3a3aee237f4b6b5 (127.1.182.130:43485)
I20260928 02:05:18.708175 9503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5edea96d6441f18980b2e7fe8c71f0" candidate_uuid: "417e8867b6b74f14a2402ef52dd71ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecd8b31074bd483e889ac06dee87dfe8" is_pre_election: true
I20260928 02:05:18.708384 9503 raft_consensus.cc:2454] T 7e5edea96d6441f18980b2e7fe8c71f0 P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 417e8867b6b74f14a2402ef52dd71ad6 in term 0.
I20260928 02:05:18.708357 9641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5edea96d6441f18980b2e7fe8c71f0" candidate_uuid: "417e8867b6b74f14a2402ef52dd71ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" is_pre_election: true
I20260928 02:05:18.708663 9641 raft_consensus.cc:2454] T 7e5edea96d6441f18980b2e7fe8c71f0 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 417e8867b6b74f14a2402ef52dd71ad6 in term 0.
I20260928 02:05:18.709014 9707 leader_election.cc:302] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 417e8867b6b74f14a2402ef52dd71ad6, ecd8b31074bd483e889ac06dee87dfe8; no voters:
I20260928 02:05:18.709350 9857 raft_consensus.cc:2781] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:05:18.709473 9857 raft_consensus.cc:482] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:05:18.709600 9857 raft_consensus.cc:3037] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:18.713133 9857 raft_consensus.cc:504] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:18.713882 9857 leader_election.cc:288] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 [CANDIDATE]: Term 1 election: Requested vote from peers ecd8b31074bd483e889ac06dee87dfe8 (127.1.182.129:35749), 9cc0277329224d2fb3a3aee237f4b6b5 (127.1.182.130:43485)
I20260928 02:05:18.714471 9503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5edea96d6441f18980b2e7fe8c71f0" candidate_uuid: "417e8867b6b74f14a2402ef52dd71ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecd8b31074bd483e889ac06dee87dfe8"
I20260928 02:05:18.714442 9641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5edea96d6441f18980b2e7fe8c71f0" candidate_uuid: "417e8867b6b74f14a2402ef52dd71ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc0277329224d2fb3a3aee237f4b6b5"
I20260928 02:05:18.714654 9503 raft_consensus.cc:3037] T 7e5edea96d6441f18980b2e7fe8c71f0 P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:18.714673 9641 raft_consensus.cc:3037] T 7e5edea96d6441f18980b2e7fe8c71f0 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:18.717525 9503 raft_consensus.cc:2454] T 7e5edea96d6441f18980b2e7fe8c71f0 P ecd8b31074bd483e889ac06dee87dfe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 417e8867b6b74f14a2402ef52dd71ad6 in term 1.
I20260928 02:05:18.718024 9707 leader_election.cc:302] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 417e8867b6b74f14a2402ef52dd71ad6, ecd8b31074bd483e889ac06dee87dfe8; no voters:
I20260928 02:05:18.718305 9908 raft_consensus.cc:2781] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:05:18.718524 9908 raft_consensus.cc:686] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 [term 1 LEADER]: Becoming Leader. State: Replica: 417e8867b6b74f14a2402ef52dd71ad6, State: Running, Role: LEADER
I20260928 02:05:18.718981 9908 consensus_queue.cc:237] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:18.718180 9641 raft_consensus.cc:2454] T 7e5edea96d6441f18980b2e7fe8c71f0 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 417e8867b6b74f14a2402ef52dd71ad6 in term 1.
I20260928 02:05:18.723444 9341 catalog_manager.cc:5495] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 417e8867b6b74f14a2402ef52dd71ad6 (127.1.182.131). New cstate: current_term: 1 leader_uuid: "417e8867b6b74f14a2402ef52dd71ad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } health_report { overall_health: HEALTHY } } }
I20260928 02:05:18.752342 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:18.746349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.1.182.131:33033)
0928 02:05:18.746586 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:18.746593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:18.746614 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:05:18.746687 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:18.746695 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:18.746842 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:18.746975 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:05:18.747364 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.748662 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.749964 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.750784 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.750946 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:18.751412 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:05:18.751563 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:05:18.751671 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:18.751764 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 02:05:18.751861 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:05:18.755936 9641 raft_consensus.cc:1261] T 7e5edea96d6441f18980b2e7fe8c71f0 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 1 FOLLOWER]: Refusing update from remote peer 417e8867b6b74f14a2402ef52dd71ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:05:18.756021 9503 raft_consensus.cc:1261] T 7e5edea96d6441f18980b2e7fe8c71f0 P ecd8b31074bd483e889ac06dee87dfe8 [term 1 FOLLOWER]: Refusing update from remote peer 417e8867b6b74f14a2402ef52dd71ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:05:18.756525 9857 consensus_queue.cc:1035] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:05:18.756791 9908 consensus_queue.cc:1035] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:05:18.776844 9341 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:05:18.777124 9341 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:05:18.780117 9341 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43: Sending DeleteTablet for 3 replicas of tablet 7e5edea96d6441f18980b2e7fe8c71f0
I20260928 02:05:18.781128 9617 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e5edea96d6441f18980b2e7fe8c71f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:05:18 UTC) from {username='oryx', principal='oryx/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:50000
I20260928 02:05:18.781131 9477 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e5edea96d6441f18980b2e7fe8c71f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:05:18 UTC) from {username='oryx', principal='oryx/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:50460
I20260928 02:05:18.781221 9756 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e5edea96d6441f18980b2e7fe8c71f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:05:18 UTC) from {username='oryx', principal='oryx/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:49008
I20260928 02:05:18.781380 9919 tablet_replica.cc:330] T 7e5edea96d6441f18980b2e7fe8c71f0 P ecd8b31074bd483e889ac06dee87dfe8: stopping tablet replica
I20260928 02:05:18.781378 9920 tablet_replica.cc:330] T 7e5edea96d6441f18980b2e7fe8c71f0 P 9cc0277329224d2fb3a3aee237f4b6b5: stopping tablet replica
I20260928 02:05:18.781570 9920 raft_consensus.cc:2229] T 7e5edea96d6441f18980b2e7fe8c71f0 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:05:18.781563 9919 raft_consensus.cc:2229] T 7e5edea96d6441f18980b2e7fe8c71f0 P ecd8b31074bd483e889ac06dee87dfe8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:05:18.781764 9919 raft_consensus.cc:2258] T 7e5edea96d6441f18980b2e7fe8c71f0 P ecd8b31074bd483e889ac06dee87dfe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:05:18.781762 9920 raft_consensus.cc:2258] T 7e5edea96d6441f18980b2e7fe8c71f0 P 9cc0277329224d2fb3a3aee237f4b6b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:05:18.782025 9921 tablet_replica.cc:330] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6: stopping tablet replica
I20260928 02:05:18.782234 9921 raft_consensus.cc:2229] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:05:18.782701 9921 raft_consensus.cc:2258] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:05:18.783239 9919 ts_tablet_manager.cc:1905] T 7e5edea96d6441f18980b2e7fe8c71f0 P ecd8b31074bd483e889ac06dee87dfe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:18.783468 9341 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:05:18.783800 9921 ts_tablet_manager.cc:1905] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 02:05:18.784170 9341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:05:18.791920 9919 ts_tablet_manager.cc:1918] T 7e5edea96d6441f18980b2e7fe8c71f0 P ecd8b31074bd483e889ac06dee87dfe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:05:18.792102 9919 log.cc:1196] T 7e5edea96d6441f18980b2e7fe8c71f0 P ecd8b31074bd483e889ac06dee87dfe8: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-0/wal/wals/7e5edea96d6441f18980b2e7fe8c71f0
I20260928 02:05:18.792155 9477 tablet_service.cc:1467] Processing CreateTablet for tablet e283e891eb884c69b939013213c5680b (DEFAULT_TABLE table=test-table [id=9d45bcf05acb45f3bceb7f5d7f4e9b73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:05:18.792510 9921 ts_tablet_manager.cc:1918] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:05:18.792569 9477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e283e891eb884c69b939013213c5680b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:18.792650 9919 ts_tablet_manager.cc:1939] T 7e5edea96d6441f18980b2e7fe8c71f0 P ecd8b31074bd483e889ac06dee87dfe8: Deleting consensus metadata
I20260928 02:05:18.793352 9921 log.cc:1196] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/7e5edea96d6441f18980b2e7fe8c71f0
I20260928 02:05:18.793865 9921 ts_tablet_manager.cc:1939] T 7e5edea96d6441f18980b2e7fe8c71f0 P 417e8867b6b74f14a2402ef52dd71ad6: Deleting consensus metadata
I20260928 02:05:18.794986 9920 ts_tablet_manager.cc:1905] T 7e5edea96d6441f18980b2e7fe8c71f0 P 9cc0277329224d2fb3a3aee237f4b6b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:18.795439 9617 tablet_service.cc:1467] Processing CreateTablet for tablet e283e891eb884c69b939013213c5680b (DEFAULT_TABLE table=test-table [id=9d45bcf05acb45f3bceb7f5d7f4e9b73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:05:18.795861 9617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e283e891eb884c69b939013213c5680b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:18.795856 9756 tablet_service.cc:1467] Processing CreateTablet for tablet e283e891eb884c69b939013213c5680b (DEFAULT_TABLE table=test-table [id=9d45bcf05acb45f3bceb7f5d7f4e9b73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:05:18.796250 9756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e283e891eb884c69b939013213c5680b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:18.799278 9325 catalog_manager.cc:4841] TS ecd8b31074bd483e889ac06dee87dfe8 (127.1.182.129:35749): tablet 7e5edea96d6441f18980b2e7fe8c71f0 (table test-table [id=db270d4a6d9c4cb49761e016d9c15667]) successfully deleted
I20260928 02:05:18.799530 9325 catalog_manager.cc:4841] TS 417e8867b6b74f14a2402ef52dd71ad6 (127.1.182.131:33033): tablet 7e5edea96d6441f18980b2e7fe8c71f0 (table test-table [id=db270d4a6d9c4cb49761e016d9c15667]) successfully deleted
I20260928 02:05:18.803174 9852 tablet_bootstrap.cc:492] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6: Bootstrap starting.
I20260928 02:05:18.803321 9920 ts_tablet_manager.cc:1918] T 7e5edea96d6441f18980b2e7fe8c71f0 P 9cc0277329224d2fb3a3aee237f4b6b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:05:18.803563 9920 log.cc:1196] T 7e5edea96d6441f18980b2e7fe8c71f0 P 9cc0277329224d2fb3a3aee237f4b6b5: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/7e5edea96d6441f18980b2e7fe8c71f0
I20260928 02:05:18.804179 9920 ts_tablet_manager.cc:1939] T 7e5edea96d6441f18980b2e7fe8c71f0 P 9cc0277329224d2fb3a3aee237f4b6b5: Deleting consensus metadata
I20260928 02:05:18.804934 9852 tablet_bootstrap.cc:654] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:18.805439 9325 catalog_manager.cc:4841] TS 9cc0277329224d2fb3a3aee237f4b6b5 (127.1.182.130:43485): tablet 7e5edea96d6441f18980b2e7fe8c71f0 (table test-table [id=db270d4a6d9c4cb49761e016d9c15667]) successfully deleted
I20260928 02:05:18.810068 9851 tablet_bootstrap.cc:492] T e283e891eb884c69b939013213c5680b P 9cc0277329224d2fb3a3aee237f4b6b5: Bootstrap starting.
I20260928 02:05:18.810827 9853 tablet_bootstrap.cc:492] T e283e891eb884c69b939013213c5680b P ecd8b31074bd483e889ac06dee87dfe8: Bootstrap starting.
I20260928 02:05:18.811868 9851 tablet_bootstrap.cc:654] T e283e891eb884c69b939013213c5680b P 9cc0277329224d2fb3a3aee237f4b6b5: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:18.813017 9853 tablet_bootstrap.cc:654] T e283e891eb884c69b939013213c5680b P ecd8b31074bd483e889ac06dee87dfe8: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:18.818289 9851 tablet_bootstrap.cc:492] T e283e891eb884c69b939013213c5680b P 9cc0277329224d2fb3a3aee237f4b6b5: No bootstrap required, opened a new log
I20260928 02:05:18.818538 9851 ts_tablet_manager.cc:1397] T e283e891eb884c69b939013213c5680b P 9cc0277329224d2fb3a3aee237f4b6b5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 02:05:18.819460 9853 tablet_bootstrap.cc:492] T e283e891eb884c69b939013213c5680b P ecd8b31074bd483e889ac06dee87dfe8: No bootstrap required, opened a new log
I20260928 02:05:18.819658 9853 ts_tablet_manager.cc:1397] T e283e891eb884c69b939013213c5680b P ecd8b31074bd483e889ac06dee87dfe8: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260928 02:05:18.819499 9851 raft_consensus.cc:348] T e283e891eb884c69b939013213c5680b P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:18.819971 9851 raft_consensus.cc:374] T e283e891eb884c69b939013213c5680b P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:18.820112 9851 raft_consensus.cc:729] T e283e891eb884c69b939013213c5680b P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc0277329224d2fb3a3aee237f4b6b5, State: Initialized, Role: FOLLOWER
I20260928 02:05:18.820462 9851 consensus_queue.cc:260] T e283e891eb884c69b939013213c5680b P 9cc0277329224d2fb3a3aee237f4b6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:18.820613 9853 raft_consensus.cc:348] T e283e891eb884c69b939013213c5680b P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:18.820917 9853 raft_consensus.cc:374] T e283e891eb884c69b939013213c5680b P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:18.820999 9853 raft_consensus.cc:729] T e283e891eb884c69b939013213c5680b P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecd8b31074bd483e889ac06dee87dfe8, State: Initialized, Role: FOLLOWER
I20260928 02:05:18.821255 9853 consensus_queue.cc:260] T e283e891eb884c69b939013213c5680b P ecd8b31074bd483e889ac06dee87dfe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:18.821858 9853 ts_tablet_manager.cc:1428] T e283e891eb884c69b939013213c5680b P ecd8b31074bd483e889ac06dee87dfe8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:05:18.821384 9851 ts_tablet_manager.cc:1428] T e283e891eb884c69b939013213c5680b P 9cc0277329224d2fb3a3aee237f4b6b5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:05:18.823465 9852 tablet_bootstrap.cc:492] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6: No bootstrap required, opened a new log
I20260928 02:05:18.823683 9852 ts_tablet_manager.cc:1397] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.011s
I20260928 02:05:18.824597 9852 raft_consensus.cc:348] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:18.824882 9852 raft_consensus.cc:374] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:18.825006 9852 raft_consensus.cc:729] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417e8867b6b74f14a2402ef52dd71ad6, State: Initialized, Role: FOLLOWER
I20260928 02:05:18.825311 9852 consensus_queue.cc:260] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:18.825802 9852 ts_tablet_manager.cc:1428] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:05:18.870783 9857 raft_consensus.cc:482] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:18.870999 9857 raft_consensus.cc:504] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:18.871592 9857 leader_election.cc:288] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecd8b31074bd483e889ac06dee87dfe8 (127.1.182.129:35749), 9cc0277329224d2fb3a3aee237f4b6b5 (127.1.182.130:43485)
I20260928 02:05:18.872174 9641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e283e891eb884c69b939013213c5680b" candidate_uuid: "417e8867b6b74f14a2402ef52dd71ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" is_pre_election: true
I20260928 02:05:18.872263 9503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e283e891eb884c69b939013213c5680b" candidate_uuid: "417e8867b6b74f14a2402ef52dd71ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecd8b31074bd483e889ac06dee87dfe8" is_pre_election: true
I20260928 02:05:18.872407 9641 raft_consensus.cc:2454] T e283e891eb884c69b939013213c5680b P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 417e8867b6b74f14a2402ef52dd71ad6 in term 0.
I20260928 02:05:18.872578 9503 raft_consensus.cc:2454] T e283e891eb884c69b939013213c5680b P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 417e8867b6b74f14a2402ef52dd71ad6 in term 0.
I20260928 02:05:18.872906 9707 leader_election.cc:302] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 417e8867b6b74f14a2402ef52dd71ad6, 9cc0277329224d2fb3a3aee237f4b6b5; no voters:
I20260928 02:05:18.873175 9857 raft_consensus.cc:2781] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:05:18.873296 9857 raft_consensus.cc:482] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:05:18.873409 9857 raft_consensus.cc:3037] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:18.875656 9857 raft_consensus.cc:504] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:18.876186 9857 leader_election.cc:288] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 [CANDIDATE]: Term 1 election: Requested vote from peers ecd8b31074bd483e889ac06dee87dfe8 (127.1.182.129:35749), 9cc0277329224d2fb3a3aee237f4b6b5 (127.1.182.130:43485)
I20260928 02:05:18.876624 9503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e283e891eb884c69b939013213c5680b" candidate_uuid: "417e8867b6b74f14a2402ef52dd71ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecd8b31074bd483e889ac06dee87dfe8"
I20260928 02:05:18.876711 9641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e283e891eb884c69b939013213c5680b" candidate_uuid: "417e8867b6b74f14a2402ef52dd71ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc0277329224d2fb3a3aee237f4b6b5"
I20260928 02:05:18.876848 9503 raft_consensus.cc:3037] T e283e891eb884c69b939013213c5680b P ecd8b31074bd483e889ac06dee87dfe8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:18.876900 9641 raft_consensus.cc:3037] T e283e891eb884c69b939013213c5680b P 9cc0277329224d2fb3a3aee237f4b6b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:18.878814 9503 raft_consensus.cc:2454] T e283e891eb884c69b939013213c5680b P ecd8b31074bd483e889ac06dee87dfe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 417e8867b6b74f14a2402ef52dd71ad6 in term 1.
I20260928 02:05:18.879392 9707 leader_election.cc:302] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 417e8867b6b74f14a2402ef52dd71ad6, ecd8b31074bd483e889ac06dee87dfe8; no voters:
I20260928 02:05:18.879410 9641 raft_consensus.cc:2454] T e283e891eb884c69b939013213c5680b P 9cc0277329224d2fb3a3aee237f4b6b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 417e8867b6b74f14a2402ef52dd71ad6 in term 1.
I20260928 02:05:18.879647 9857 raft_consensus.cc:2781] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:05:18.879830 9857 raft_consensus.cc:686] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 [term 1 LEADER]: Becoming Leader. State: Replica: 417e8867b6b74f14a2402ef52dd71ad6, State: Running, Role: LEADER
I20260928 02:05:18.880118 9857 consensus_queue.cc:237] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } }
I20260928 02:05:18.882675 9339 catalog_manager.cc:5495] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 417e8867b6b74f14a2402ef52dd71ad6 (127.1.182.131). New cstate: current_term: 1 leader_uuid: "417e8867b6b74f14a2402ef52dd71ad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "417e8867b6b74f14a2402ef52dd71ad6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33033 } health_report { overall_health: HEALTHY } } }
I20260928 02:05:18.910930 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:18.905523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:53013 (local address 127.1.182.131:33033)
0928 02:05:18.905681 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:18.905685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:18.905699 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:05:18.905888 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:18.905891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:18.905959 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:18.906029 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:05:18.906443 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.907222 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.909147 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.909699 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.909788 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:18.910227 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:05:18.910335 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:05:18.910448 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:18.910598 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 02:05:18.910694 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:05:18.911164 9916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:18.905439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:33033 (local address 127.1.182.131:53013)
0928 02:05:18.905629 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:18.905651 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:18.905763 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:18.906083 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:18.906089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:18.906101 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:05:18.906301 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:18.906328 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.907355 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.907362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:18.909022 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:18.909037 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.909834 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.909841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:18.910033 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:18.910087 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:05:18.910475 (+ 388us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:05:18.910487 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:05:18.910583 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 02:05:18.910751 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 02:05:18.918695 9641 raft_consensus.cc:1261] T e283e891eb884c69b939013213c5680b P 9cc0277329224d2fb3a3aee237f4b6b5 [term 1 FOLLOWER]: Refusing update from remote peer 417e8867b6b74f14a2402ef52dd71ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:05:18.918919 9503 raft_consensus.cc:1261] T e283e891eb884c69b939013213c5680b P ecd8b31074bd483e889ac06dee87dfe8 [term 1 FOLLOWER]: Refusing update from remote peer 417e8867b6b74f14a2402ef52dd71ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:05:18.919245 9908 consensus_queue.cc:1035] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cc0277329224d2fb3a3aee237f4b6b5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:05:18.919548 9857 consensus_queue.cc:1035] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecd8b31074bd483e889ac06dee87dfe8" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:05:18.970579 9339 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:05:18.970853 9339 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:05:18.973775 9339 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87050c7ca1a14e17982d9ffa888c5e43: Sending DeleteTablet for 3 replicas of tablet e283e891eb884c69b939013213c5680b
I20260928 02:05:18.974525 9477 tablet_service.cc:1514] Processing DeleteTablet for tablet e283e891eb884c69b939013213c5680b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:05:18 UTC) from {username='oryx', principal='oryx/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:50460
I20260928 02:05:18.974587 9617 tablet_service.cc:1514] Processing DeleteTablet for tablet e283e891eb884c69b939013213c5680b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:05:18 UTC) from {username='oryx', principal='oryx/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:50000
I20260928 02:05:18.974848 9919 tablet_replica.cc:330] T e283e891eb884c69b939013213c5680b P ecd8b31074bd483e889ac06dee87dfe8: stopping tablet replica
I20260928 02:05:18.974871 9756 tablet_service.cc:1514] Processing DeleteTablet for tablet e283e891eb884c69b939013213c5680b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:05:18 UTC) from {username='oryx', principal='oryx/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:49008
I20260928 02:05:18.975055 9919 raft_consensus.cc:2229] T e283e891eb884c69b939013213c5680b P ecd8b31074bd483e889ac06dee87dfe8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:05:18.975165 9921 tablet_replica.cc:330] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6: stopping tablet replica
I20260928 02:05:18.975325 9919 raft_consensus.cc:2258] T e283e891eb884c69b939013213c5680b P ecd8b31074bd483e889ac06dee87dfe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:05:18.975368 9921 raft_consensus.cc:2229] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:05:18.975708 9921 raft_consensus.cc:2258] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:05:18.976236 9919 ts_tablet_manager.cc:1905] T e283e891eb884c69b939013213c5680b P ecd8b31074bd483e889ac06dee87dfe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:18.976939 9921 ts_tablet_manager.cc:1905] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:18.978709 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 9412
I20260928 02:05:18.975008 9920 tablet_replica.cc:330] T e283e891eb884c69b939013213c5680b P 9cc0277329224d2fb3a3aee237f4b6b5: stopping tablet replica
I20260928 02:05:18.979532 9920 raft_consensus.cc:2229] T e283e891eb884c69b939013213c5680b P 9cc0277329224d2fb3a3aee237f4b6b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:05:18.979830 9920 raft_consensus.cc:2258] T e283e891eb884c69b939013213c5680b P 9cc0277329224d2fb3a3aee237f4b6b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:05:18.981165 9920 ts_tablet_manager.cc:1905] T e283e891eb884c69b939013213c5680b P 9cc0277329224d2fb3a3aee237f4b6b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:18.983234 9919 ts_tablet_manager.cc:1918] T e283e891eb884c69b939013213c5680b P ecd8b31074bd483e889ac06dee87dfe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:05:18.983417 9919 log.cc:1196] T e283e891eb884c69b939013213c5680b P ecd8b31074bd483e889ac06dee87dfe8: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-0/wal/wals/e283e891eb884c69b939013213c5680b
I20260928 02:05:18.983851 9919 ts_tablet_manager.cc:1939] T e283e891eb884c69b939013213c5680b P ecd8b31074bd483e889ac06dee87dfe8: Deleting consensus metadata
I20260928 02:05:18.984903 9325 catalog_manager.cc:4841] TS ecd8b31074bd483e889ac06dee87dfe8 (127.1.182.129:35749): tablet e283e891eb884c69b939013213c5680b (table test-table [id=9d45bcf05acb45f3bceb7f5d7f4e9b73]) successfully deleted
I20260928 02:05:18.988637 9921 ts_tablet_manager.cc:1918] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:05:18.988792 9921 log.cc:1196] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/e283e891eb884c69b939013213c5680b
I20260928 02:05:18.988831 9920 ts_tablet_manager.cc:1918] T e283e891eb884c69b939013213c5680b P 9cc0277329224d2fb3a3aee237f4b6b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:05:18.989002 9920 log.cc:1196] T e283e891eb884c69b939013213c5680b P 9cc0277329224d2fb3a3aee237f4b6b5: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/e283e891eb884c69b939013213c5680b
I20260928 02:05:18.989276 9921 ts_tablet_manager.cc:1939] T e283e891eb884c69b939013213c5680b P 417e8867b6b74f14a2402ef52dd71ad6: Deleting consensus metadata
I20260928 02:05:18.989470 9920 ts_tablet_manager.cc:1939] T e283e891eb884c69b939013213c5680b P 9cc0277329224d2fb3a3aee237f4b6b5: Deleting consensus metadata
I20260928 02:05:18.990504 9325 catalog_manager.cc:4841] TS 9cc0277329224d2fb3a3aee237f4b6b5 (127.1.182.130:43485): tablet e283e891eb884c69b939013213c5680b (table test-table [id=9d45bcf05acb45f3bceb7f5d7f4e9b73]) successfully deleted
I20260928 02:05:18.990778 9325 catalog_manager.cc:4841] TS 417e8867b6b74f14a2402ef52dd71ad6 (127.1.182.131:33033): tablet e283e891eb884c69b939013213c5680b (table test-table [id=9d45bcf05acb45f3bceb7f5d7f4e9b73]) successfully deleted
Sep 28 02:05:18 dist-test-slave-23bp krb5kdc[9296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.182.129@KRBTEST.COM
I20260928 02:05:19.006448 9850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:18.979924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54462 (local address 127.1.182.129:35749)
0928 02:05:18.980103 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:18.980109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:18.991115 (+ 11006us) server_negotiation.cc:407] Connection header received
0928 02:05:18.991322 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:18.991327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:18.991405 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:18.991517 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:18.992653 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.993743 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.994910 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:18.995595 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:18.995722 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:18.999893 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:18.999911 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:18.999933 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:18.999980 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:19.003773 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:19.004423 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:19.004432 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:19.004436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:19.004594 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:19.004909 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:19.004916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:19.004919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:19.005362 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:19.005660 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:19.005838 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 02:05:19.006061 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:05:19.007221 9543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:19.130771 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 9412
I20260928 02:05:19.153961 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 9552
Sep 28 02:05:19 dist-test-slave-23bp krb5kdc[9296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.182.130@KRBTEST.COM
I20260928 02:05:19.169513 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:19.154819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33870 (local address 127.1.182.130:43485)
0928 02:05:19.155354 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:19.155358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:19.155373 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:05:19.155425 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:19.155428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:19.155490 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:19.155613 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:19.156767 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:19.157482 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:19.158698 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:19.159249 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:19.159380 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:19.163830 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:19.163842 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:19.163858 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:19.163897 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:19.166757 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:19.167490 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:19.167497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:19.167500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:19.167625 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:19.168007 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:19.168016 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:19.168018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:19.168350 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:19.168641 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:19.168939 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 02:05:19.169169 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":266,"threads_started":1}
I20260928 02:05:19.170243 9682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:19.288136 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 9552
I20260928 02:05:19.306751 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 9691
Sep 28 02:05:19 dist-test-slave-23bp krb5kdc[9296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.182.131@KRBTEST.COM
I20260928 02:05:19.323575 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:19.307733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42436 (local address 127.1.182.131:33033)
0928 02:05:19.307908 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:19.307912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:19.307988 (+ 76us) server_negotiation.cc:407] Connection header received
0928 02:05:19.308158 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:19.308162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:19.308222 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:19.308311 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:19.309926 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:19.310656 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:19.312402 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:19.312893 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:19.313029 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:19.318114 (+ 5085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:19.318130 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:19.318146 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:19.318193 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:19.321086 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:19.321701 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:19.321708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:19.321710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:19.321835 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:19.322134 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:19.322137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:19.322139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:19.322468 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:19.322684 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:19.322898 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 02:05:19.323046 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:05:19.324273 9821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:19.468075 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 9691
I20260928 02:05:19.493932 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 9309
I20260928 02:05:19.495196 9381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:19.610809 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 9309
2026-09-28T02:05:19Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6790 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:05:19 dist-test-slave-23bp krb5kdc[9957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:05:19 dist-test-slave-23bp krb5kdc[9957](info): set up 2 sockets
Sep 28 02:05:19 dist-test-slave-23bp krb5kdc[9957](info): commencing operation
krb5kdc: starting...
W20260928 02:05:20.730473 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:05:20 dist-test-slave-23bp krb5kdc[9957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:05:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:05:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:05:20.950070 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:45039
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:42221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:45039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:21.275710 9970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:21.275988 9970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:21.276160 9970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:21.283432 9970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:05:21.283548 9970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:21.283630 9970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:21.283736 9970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:05:21.283826 9970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:05:21.293592 9970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:42221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:45039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:45039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:21.294934 9970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:21.296319 9970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:21.308859 9976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:21.308984 9975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:21.308883 9978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:21.309576 9970 server_base.cc:1034] running on GCE node
I20260928 02:05:21.310415 9970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:21.311865 9970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:21.313117 9970 hybrid_clock.cc:648] HybridClock initialized: now 1790561121313063 us; error 48 us; skew 500 ppm
I20260928 02:05:21.313462 9970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T02:05:21Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1769 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:05:21 dist-test-slave-23bp krb5kdc[9984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:05:21 dist-test-slave-23bp krb5kdc[9984](info): set up 2 sockets
Sep 28 02:05:21 dist-test-slave-23bp krb5kdc[9984](info): commencing operation
krb5kdc: starting...
W20260928 02:05:22.474364 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.008s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:05:22 dist-test-slave-23bp krb5kdc[9984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:05:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:05:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:05:22.708807 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:33023
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:45099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:33023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:23.022998 9997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:23.023378 9997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:23.023540 9997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:23.030593 9997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:05:23.030741 9997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:23.030857 9997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:23.030962 9997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:05:23.031052 9997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:05:23.040490 9997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:45099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:33023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:33023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:23.041783 9997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:23.043092 9997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:23.054467 10002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:23.054514 10005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:23.054554 10003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:23.055486 9997 server_base.cc:1034] running on GCE node
I20260928 02:05:23.056195 9997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:23.057502 9997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:23.058720 9997 hybrid_clock.cc:648] HybridClock initialized: now 1790561123058690 us; error 42 us; skew 500 ppm
I20260928 02:05:23.059052 9997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:23 dist-test-slave-23bp krb5kdc[9984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561123, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:23.062252 9997 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:05:23.063597 9997 webserver.cc:460] Webserver started at http://127.1.182.190:41569/ using document root <none> and password file <none>
I20260928 02:05:23.064198 9997 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:23.064288 9997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:23.064538 9997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:23.067360 9997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "d0c1be5dcb0542fd81c6ada882970e30"
format_stamp: "Formatted at 2026-09-28 02:05:23 on dist-test-slave-23bp"
server_key: "d5b08558a55a8abab0cc4035e3d543c2"
server_key_iv: "0753679acb6df8500327182abdf24fb0"
server_key_version: "encryptionkey@0"
I20260928 02:05:23.068064 9997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "d0c1be5dcb0542fd81c6ada882970e30"
format_stamp: "Formatted at 2026-09-28 02:05:23 on dist-test-slave-23bp"
server_key: "d5b08558a55a8abab0cc4035e3d543c2"
server_key_iv: "0753679acb6df8500327182abdf24fb0"
server_key_version: "encryptionkey@0"
I20260928 02:05:23.072777 9997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 02:05:23.075914 10012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:23.077136 9997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:05:23.077308 9997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "d0c1be5dcb0542fd81c6ada882970e30"
format_stamp: "Formatted at 2026-09-28 02:05:23 on dist-test-slave-23bp"
server_key: "d5b08558a55a8abab0cc4035e3d543c2"
server_key_iv: "0753679acb6df8500327182abdf24fb0"
server_key_version: "encryptionkey@0"
I20260928 02:05:23.077502 9997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:23.110353 9997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:23.114306 9997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:23.114600 9997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:23.132652 9997 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:33023
I20260928 02:05:23.132689 10063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:33023 every 8 connection(s)
I20260928 02:05:23.134054 9997 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:05:23.137956 10064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:23.138396 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 9997
I20260928 02:05:23.138607 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:05:23.139287 1754 external_mini_cluster.cc:1428] Setting key ff9aaf728f70a0909ae66a1fc9ff69e8
I20260928 02:05:23.152724 10064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30: Bootstrap starting.
Sep 28 02:05:23 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:05:23.156745 10064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:23.158272 10064 log.cc:824] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30: Log is configured to *not* fsync() on all Append() calls
I20260928 02:05:23.161813 10064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30: No bootstrap required, opened a new log
I20260928 02:05:23.164641 10067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:23.141203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45224 (local address 127.1.182.190:33023)
0928 02:05:23.141959 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:23.141971 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:23.142014 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:05:23.144133 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:23.144160 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:23.144568 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:23.144903 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:23.146880 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:23.148250 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:23.150229 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:23.150882 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:23.151011 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:23.155850 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:23.155863 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:23.155883 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:23.155939 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:23.159982 (+ 4043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:23.160753 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:23.160762 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:23.160770 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:23.160999 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:23.161346 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:23.161351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:23.161355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:23.162064 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:23.162400 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:23.162695 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 02:05:23.163174 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":557,"thread_start_us":358,"threads_started":1}
I20260928 02:05:23.171394 10064 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c1be5dcb0542fd81c6ada882970e30" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33023 } }
I20260928 02:05:23.171753 10064 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:23.171828 10064 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0c1be5dcb0542fd81c6ada882970e30, State: Initialized, Role: FOLLOWER
I20260928 02:05:23.172475 10064 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c1be5dcb0542fd81c6ada882970e30" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33023 } }
I20260928 02:05:23.172791 10064 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:05:23.172942 10064 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:05:23.173137 10064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:23.175570 10064 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c1be5dcb0542fd81c6ada882970e30" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33023 } }
I20260928 02:05:23.176072 10064 leader_election.cc:302] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0c1be5dcb0542fd81c6ada882970e30; no voters:
I20260928 02:05:23.176793 10064 leader_election.cc:288] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:05:23.176971 10069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:05:23.177630 10069 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [term 1 LEADER]: Becoming Leader. State: Replica: d0c1be5dcb0542fd81c6ada882970e30, State: Running, Role: LEADER
I20260928 02:05:23.178195 10069 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c1be5dcb0542fd81c6ada882970e30" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33023 } }
I20260928 02:05:23.179152 10064 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:05:23.181355 10070 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0c1be5dcb0542fd81c6ada882970e30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c1be5dcb0542fd81c6ada882970e30" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33023 } } }
I20260928 02:05:23.181893 10070 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:23.182410 10071 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0c1be5dcb0542fd81c6ada882970e30. Latest consensus state: current_term: 1 leader_uuid: "d0c1be5dcb0542fd81c6ada882970e30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c1be5dcb0542fd81c6ada882970e30" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33023 } } }
I20260928 02:05:23.182699 10071 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:23.184485 10077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:05:23.189626 10077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:05:23.198392 10077 catalog_manager.cc:1347] Generated new cluster ID: 12dd774fb9f949a4b35dc6072b179a3b
I20260928 02:05:23.198513 10077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:05:23.211977 10077 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:05:23.213836 10077 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:05:23.232437 10077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30: Generated new TSK 0
I20260928 02:05:23.233331 10077 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.182.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
I20260928 02:05:23.293470 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:33023
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:45099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:23.620039 10090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:23.620369 10090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:23.620528 10090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:23.627653 10090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:23.627802 10090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:23.627993 10090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:05:23.637562 10090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:45099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:33023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:23.638840 10090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:23.640131 10090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:23.652787 10098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:23.652827 10096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:23.652858 10095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:23.653932 10090 server_base.cc:1034] running on GCE node
I20260928 02:05:23.654526 10090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:23.655526 10090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:23.656765 10090 hybrid_clock.cc:648] HybridClock initialized: now 1790561123656724 us; error 48 us; skew 500 ppm
I20260928 02:05:23.657092 10090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:23 dist-test-slave-23bp krb5kdc[9984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561123, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:23.660152 10090 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:05:23.661584 10090 webserver.cc:460] Webserver started at http://127.1.182.129:41623/ using document root <none> and password file <none>
I20260928 02:05:23.662263 10090 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:23.662397 10090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:23.662693 10090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:23.665930 10090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "af998492a38d4b208c4a5fac8848a60a"
format_stamp: "Formatted at 2026-09-28 02:05:23 on dist-test-slave-23bp"
server_key: "6dea08487d50a382a4b94c5522d184bc"
server_key_iv: "c38cb79497a673f0b4f004383dc3a83a"
server_key_version: "encryptionkey@0"
I20260928 02:05:23.666719 10090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "af998492a38d4b208c4a5fac8848a60a"
format_stamp: "Formatted at 2026-09-28 02:05:23 on dist-test-slave-23bp"
server_key: "6dea08487d50a382a4b94c5522d184bc"
server_key_iv: "c38cb79497a673f0b4f004383dc3a83a"
server_key_version: "encryptionkey@0"
I20260928 02:05:23.671782 10090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:05:23.675719 10105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:23.677192 10090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:05:23.677405 10090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "af998492a38d4b208c4a5fac8848a60a"
format_stamp: "Formatted at 2026-09-28 02:05:23 on dist-test-slave-23bp"
server_key: "6dea08487d50a382a4b94c5522d184bc"
server_key_iv: "c38cb79497a673f0b4f004383dc3a83a"
server_key_version: "encryptionkey@0"
I20260928 02:05:23.677623 10090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:23.696856 10090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:23.700976 10090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:23.701282 10090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:23.702260 10090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:23.703744 10090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:23.703853 10090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:23.704015 10090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:23.704093 10090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:23.738720 10090 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:40179
I20260928 02:05:23.738755 10218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:40179 every 8 connection(s)
I20260928 02:05:23.740127 10090 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:05:23.747495 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 10090
I20260928 02:05:23.747680 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:05:23.748317 1754 external_mini_cluster.cc:1428] Setting key 47c02262577a89a88e93667f08fbae96
Sep 28 02:05:23 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561123, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:05:23.761211 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:23.742647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:53549 (local address 127.1.182.190:33023)
0928 02:05:23.743156 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:23.743167 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:23.744143 (+ 976us) server_negotiation.cc:407] Connection header received
0928 02:05:23.746052 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:23.746058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:23.746163 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:23.746293 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:23.748120 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:23.748969 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:23.750923 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:23.751431 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:23.751578 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:23.755165 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:23.755170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:23.755174 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:23.755204 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:23.757902 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:23.758668 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:23.758675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:23.758679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:23.758808 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:23.759315 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:23.759322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:23.759327 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:23.759682 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:23.759836 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:23.760715 (+ 879us) server_negotiation.cc:299] Negotiation successful
0928 02:05:23.760915 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":234,"threads_started":1}
I20260928 02:05:23.762619 10222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:23.743261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.129:53549)
0928 02:05:23.743971 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:23.744018 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:23.745773 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:23.746590 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:23.746599 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:23.747235 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:23.747942 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:23.747961 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:23.749137 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:23.749145 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:23.750764 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:23.750776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:23.751667 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:23.751677 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:23.751943 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:23.752633 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:23.752655 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:23.754924 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:23.758064 (+ 3140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:23.758074 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:23.758090 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:23.758542 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:23.758919 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:23.758926 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:23.758930 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:23.759202 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:23.759829 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:23.759839 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:23.760292 (+ 453us) client_negotiation.cc:769] Sending connection context
0928 02:05:23.760595 (+ 303us) client_negotiation.cc:240] Negotiation successful
0928 02:05:23.761188 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":274,"threads_started":1}
I20260928 02:05:23.764277 10219 heartbeater.cc:347] Connected to a master server at 127.1.182.190:33023
I20260928 02:05:23.764587 10219 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:23.765292 10219 heartbeater.cc:511] Master 127.1.182.190:33023 requested a full tablet report, sending...
I20260928 02:05:23.767364 10029 ts_manager.cc:194] Registered new tserver with Master: af998492a38d4b208c4a5fac8848a60a (127.1.182.129:40179)
I20260928 02:05:23.768750 10029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:53549
WARNING: no policy specified for kudu/127.1.182.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
I20260928 02:05:23.797390 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:33023
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:45099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:24.114327 10225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:24.114612 10225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:24.114705 10225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:24.121826 10225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:24.121933 10225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:24.122043 10225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:05:24.131402 10225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:45099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:33023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:24.132623 10225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:24.133935 10225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:24.149374 10230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:24.149322 10233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:24.149421 10231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:24.152410 10225 server_base.cc:1034] running on GCE node
I20260928 02:05:24.152988 10225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:24.153868 10225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:24.155058 10225 hybrid_clock.cc:648] HybridClock initialized: now 1790561124155016 us; error 31 us; skew 500 ppm
I20260928 02:05:24.155429 10225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:24 dist-test-slave-23bp krb5kdc[9984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561124, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:24.158345 10225 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:05:24.159844 10225 webserver.cc:460] Webserver started at http://127.1.182.130:42117/ using document root <none> and password file <none>
I20260928 02:05:24.160594 10225 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:24.160720 10225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:24.161031 10225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:24.163944 10225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "23cb00a9a39a4bb89a4cdeba777a11bb"
format_stamp: "Formatted at 2026-09-28 02:05:24 on dist-test-slave-23bp"
server_key: "9f3825b96eba0c0866a502e4f2df5778"
server_key_iv: "168b20e32b6530e44c6141755f63e384"
server_key_version: "encryptionkey@0"
I20260928 02:05:24.164712 10225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "23cb00a9a39a4bb89a4cdeba777a11bb"
format_stamp: "Formatted at 2026-09-28 02:05:24 on dist-test-slave-23bp"
server_key: "9f3825b96eba0c0866a502e4f2df5778"
server_key_iv: "168b20e32b6530e44c6141755f63e384"
server_key_version: "encryptionkey@0"
I20260928 02:05:24.169690 10225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:05:24.173026 10240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:24.174288 10225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:05:24.174470 10225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "23cb00a9a39a4bb89a4cdeba777a11bb"
format_stamp: "Formatted at 2026-09-28 02:05:24 on dist-test-slave-23bp"
server_key: "9f3825b96eba0c0866a502e4f2df5778"
server_key_iv: "168b20e32b6530e44c6141755f63e384"
server_key_version: "encryptionkey@0"
I20260928 02:05:24.174664 10225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:24.195993 10225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:24.199721 10225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:24.199946 10225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:24.200800 10225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:24.202039 10225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:24.202109 10225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:24.202188 10225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:24.202231 10225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:24.234095 10225 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:40979
I20260928 02:05:24.234184 10354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:40979 every 8 connection(s)
I20260928 02:05:24.235667 10225 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:05:24.238072 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 10225
I20260928 02:05:24.238251 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:05:24.238718 1754 external_mini_cluster.cc:1428] Setting key b5120f93449026224c8f28ced8f57d52
Sep 28 02:05:24 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561124, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:05:24.257952 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:24.238127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:56071 (local address 127.1.182.190:33023)
0928 02:05:24.239267 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:24.239274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:24.240282 (+ 1008us) server_negotiation.cc:407] Connection header received
0928 02:05:24.241661 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:24.241670 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:24.241760 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:24.241869 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:24.243358 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:24.244481 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:24.246402 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:24.247174 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:24.247316 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:24.251642 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:24.251647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:24.251651 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:24.251685 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:24.254505 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:24.255498 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:24.255503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:24.255505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:24.255638 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:24.256147 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:24.256151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:24.256153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:24.256417 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:24.256541 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:24.257531 (+ 990us) server_negotiation.cc:299] Negotiation successful
0928 02:05:24.257720 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":996}
I20260928 02:05:24.259546 10357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:24.238661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.130:56071)
0928 02:05:24.239519 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:24.239562 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:24.241393 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:24.242085 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:24.242093 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:24.242572 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:24.243200 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:24.243219 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:24.244668 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:24.244676 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:24.246263 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:24.246275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:24.247394 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:24.247402 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:24.247744 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:24.248472 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:24.248506 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:24.251398 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:24.254643 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:24.254655 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:24.254674 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:24.255357 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:24.255753 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:24.255761 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:24.255767 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:24.256007 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:24.256570 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:24.256581 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:24.257051 (+ 470us) client_negotiation.cc:769] Sending connection context
0928 02:05:24.257361 (+ 310us) client_negotiation.cc:240] Negotiation successful
0928 02:05:24.257964 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":391,"threads_started":1}
I20260928 02:05:24.261023 10355 heartbeater.cc:347] Connected to a master server at 127.1.182.190:33023
I20260928 02:05:24.261339 10355 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.182.131@KRBTEST.COM; defaulting to no policy
I20260928 02:05:24.262105 10355 heartbeater.cc:511] Master 127.1.182.190:33023 requested a full tablet report, sending...
I20260928 02:05:24.263588 10029 ts_manager.cc:194] Registered new tserver with Master: 23cb00a9a39a4bb89a4cdeba777a11bb (127.1.182.130:40979)
I20260928 02:05:24.264331 10029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:56071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
I20260928 02:05:24.290232 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:33023
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:45099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:24.606786 10360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:24.607147 10360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:24.607322 10360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:24.614436 10360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:24.614578 10360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:24.614769 10360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:05:24.624593 10360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:45099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:33023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:24.625883 10360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:24.627254 10360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:24.639983 10365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:24.639983 10368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:24.640020 10366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:24.641299 10360 server_base.cc:1034] running on GCE node
I20260928 02:05:24.641892 10360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:24.642833 10360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:24.644054 10360 hybrid_clock.cc:648] HybridClock initialized: now 1790561124643998 us; error 70 us; skew 500 ppm
I20260928 02:05:24.644368 10360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:24 dist-test-slave-23bp krb5kdc[9984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561124, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:24.647480 10360 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:05:24.648912 10360 webserver.cc:460] Webserver started at http://127.1.182.131:43567/ using document root <none> and password file <none>
I20260928 02:05:24.649626 10360 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:24.649765 10360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:24.650074 10360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:24.652911 10360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "23d534de6bd84592bcc63ced298c73e1"
format_stamp: "Formatted at 2026-09-28 02:05:24 on dist-test-slave-23bp"
server_key: "21d90bc7b238d59dcd1a9a3999600718"
server_key_iv: "0031f5cf9a5aca3d611b2a80b1178bc3"
server_key_version: "encryptionkey@0"
I20260928 02:05:24.653662 10360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "23d534de6bd84592bcc63ced298c73e1"
format_stamp: "Formatted at 2026-09-28 02:05:24 on dist-test-slave-23bp"
server_key: "21d90bc7b238d59dcd1a9a3999600718"
server_key_iv: "0031f5cf9a5aca3d611b2a80b1178bc3"
server_key_version: "encryptionkey@0"
I20260928 02:05:24.658452 10360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:05:24.661708 10375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:24.663285 10360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:05:24.663463 10360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "23d534de6bd84592bcc63ced298c73e1"
format_stamp: "Formatted at 2026-09-28 02:05:24 on dist-test-slave-23bp"
server_key: "21d90bc7b238d59dcd1a9a3999600718"
server_key_iv: "0031f5cf9a5aca3d611b2a80b1178bc3"
server_key_version: "encryptionkey@0"
I20260928 02:05:24.663645 10360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:24.681881 10360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:24.686367 10360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:24.686663 10360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:24.687608 10360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:24.688884 10360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:24.688987 10360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:24.689139 10360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:24.689211 10360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:24.721844 10360 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:44457
I20260928 02:05:24.721890 10488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:44457 every 8 connection(s)
I20260928 02:05:24.723176 10360 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:05:24.730940 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 10360
I20260928 02:05:24.731163 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:05:24.731598 1754 external_mini_cluster.cc:1428] Setting key 0bf321ed9812ffb7e730b013b34a2d32
Sep 28 02:05:24 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561124, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:05:24.743515 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:24.725709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:38043 (local address 127.1.182.190:33023)
0928 02:05:24.725883 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:24.725888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:24.727098 (+ 1210us) server_negotiation.cc:407] Connection header received
0928 02:05:24.728906 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:24.728913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:24.729004 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:24.729114 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:24.730633 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:24.731571 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:24.733318 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:24.733771 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:24.733909 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:24.737645 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:24.737650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:24.737658 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:24.737691 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:24.740295 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:24.741183 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:24.741186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:24.741188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:24.741307 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:24.741813 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:24.741816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:24.741818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:24.742120 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:24.742248 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:24.743113 (+ 865us) server_negotiation.cc:299] Negotiation successful
0928 02:05:24.743279 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:05:24.744987 10491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:24.726227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.131:38043)
0928 02:05:24.726909 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:24.726970 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:24.728621 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:24.729349 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:24.729357 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:24.729856 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:24.730451 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:24.730469 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:24.731732 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:24.731737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:24.733183 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:24.733194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:24.734000 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:24.734009 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:24.734254 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:24.734995 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:24.735025 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:24.737457 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:24.740445 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:24.740455 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:24.740472 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:24.741058 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:24.741422 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:24.741428 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:24.741432 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:24.741685 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:24.742270 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:24.742279 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:24.742689 (+ 410us) client_negotiation.cc:769] Sending connection context
0928 02:05:24.743000 (+ 311us) client_negotiation.cc:240] Negotiation successful
0928 02:05:24.743588 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":268,"threads_started":1}
I20260928 02:05:24.746382 10489 heartbeater.cc:347] Connected to a master server at 127.1.182.190:33023
I20260928 02:05:24.746711 10489 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:24.747466 10489 heartbeater.cc:511] Master 127.1.182.190:33023 requested a full tablet report, sending...
I20260928 02:05:24.748816 10029 ts_manager.cc:194] Registered new tserver with Master: 23d534de6bd84592bcc63ced298c73e1 (127.1.182.131:44457)
I20260928 02:05:24.749588 10029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:38043
I20260928 02:05:24.758634 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:05:24.771621 10219 heartbeater.cc:503] Master 127.1.182.190:33023 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.1.182.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.182.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.182.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.129.keytab.
Entry for principal oryx/127.1.182.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.129.keytab.
WARNING: no policy specified for oryx/127.1.182.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.182.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.182.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.130.keytab.
Entry for principal oryx/127.1.182.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.130.keytab.
WARNING: no policy specified for oryx/127.1.182.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.182.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.182.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.131.keytab.
Entry for principal oryx/127.1.182.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.131.keytab.
I20260928 02:05:24.895694 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 10090
Sep 28 02:05:24 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:05:24.911206 10499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:24.896828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58932 (local address 127.1.182.129:40179)
0928 02:05:24.897489 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:24.897496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:24.897523 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:05:24.897583 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:24.897590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:24.897800 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:24.897945 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:24.899252 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:24.900255 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:24.901406 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:24.901961 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:24.902056 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:24.906036 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:24.906048 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:24.906064 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:24.906096 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:24.908781 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:24.909440 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:24.909447 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:24.909450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:24.909566 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:24.909869 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:24.909872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:24.909874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:24.910194 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:24.910408 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:24.910652 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 02:05:24.910844 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":353,"threads_started":1}
I20260928 02:05:24.912454 10214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:24.994815 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 10090
I20260928 02:05:25.010980 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 10225
Sep 28 02:05:25 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.130@KRBTEST.COM
I20260928 02:05:25.026678 10501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:25.011880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.1.182.130:40979)
0928 02:05:25.012442 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:25.012455 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:25.012486 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:05:25.012561 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:25.012571 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:25.012842 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:25.013020 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:25.014254 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:25.015321 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:25.016583 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:25.017112 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:25.017248 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:25.021325 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:25.021335 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:25.021348 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:25.021393 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:25.024005 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:25.024651 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:25.024658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:25.024660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:25.024777 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:25.025149 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:25.025161 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:25.025163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:25.025487 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:25.025793 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:25.026075 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 02:05:25.026330 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":266,"threads_started":1}
I20260928 02:05:25.028071 10350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:25.109666 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 10225
I20260928 02:05:25.125195 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 10360
Sep 28 02:05:25 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.131@KRBTEST.COM
I20260928 02:05:25.141752 10503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:25.126249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58222 (local address 127.1.182.131:44457)
0928 02:05:25.126869 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:25.126876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:25.126905 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:05:25.126977 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:25.126984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:25.127263 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:25.127396 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:25.128610 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:25.129609 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:25.130766 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:25.131436 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:25.131535 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:25.135715 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:25.135729 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:25.135746 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:25.135786 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:25.139312 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:25.139888 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:25.139897 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:25.139901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:25.140049 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:25.140357 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:25.140362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:25.140366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:25.140745 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:25.141058 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:25.141184 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 02:05:25.141363 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":330,"threads_started":1}
I20260928 02:05:25.142984 10484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:25.225912 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 10360
I20260928 02:05:25.242062 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 9997
I20260928 02:05:25.243353 10059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:25.345970 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 9997
I20260928 02:05:25.366144 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:33023
--webserver_interface=127.1.182.190
--webserver_port=41569
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:45099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:33023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:25.687270 10506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:25.687600 10506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:25.687753 10506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:25.694844 10506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:05:25.694990 10506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:25.695117 10506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:25.695219 10506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:05:25.695309 10506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:05:25.708086 10506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:45099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:33023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:33023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=41569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:25.709362 10506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:25.710601 10506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:25.722338 10512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:25.722388 10511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:25.722306 10514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:25.723153 10506 server_base.cc:1034] running on GCE node
I20260928 02:05:25.723850 10506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:25.725149 10506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:25.726377 10506 hybrid_clock.cc:648] HybridClock initialized: now 1790561125726326 us; error 61 us; skew 500 ppm
I20260928 02:05:25.726759 10506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:25 dist-test-slave-23bp krb5kdc[9984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561125, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:25.729700 10506 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:05:25.731163 10506 webserver.cc:460] Webserver started at http://127.1.182.190:41569/ using document root <none> and password file <none>
I20260928 02:05:25.731820 10506 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:25.731977 10506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:25.737154 10506 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 02:05:25.739804 10521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:25.741047 10506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:05:25.741221 10506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "d0c1be5dcb0542fd81c6ada882970e30"
format_stamp: "Formatted at 2026-09-28 02:05:23 on dist-test-slave-23bp"
server_key: "d5b08558a55a8abab0cc4035e3d543c2"
server_key_iv: "0753679acb6df8500327182abdf24fb0"
server_key_version: "encryptionkey@0"
I20260928 02:05:25.742122 10506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:25.767776 10506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:25.771516 10506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:25.771775 10506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:25.789528 10506 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:33023
I20260928 02:05:25.789548 10572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:33023 every 8 connection(s)
I20260928 02:05:25.790974 10506 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:05:25.797328 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 10506
I20260928 02:05:25.798090 10573 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:05:25.798310 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:40179
--local_ip_for_outbound_sockets=127.1.182.129
--tserver_master_addrs=127.1.182.190:33023
--webserver_port=41623
--webserver_interface=127.1.182.129
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:45099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.182.129
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.129.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:05:25.803182 10573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30: Bootstrap starting.
I20260928 02:05:25.822579 10573 log.cc:824] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30: Log is configured to *not* fsync() on all Append() calls
I20260928 02:05:25.831032 10573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:05:25.831609 10573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30: Bootstrap complete.
I20260928 02:05:25.840914 10573 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c1be5dcb0542fd81c6ada882970e30" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33023 } }
I20260928 02:05:25.841672 10573 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0c1be5dcb0542fd81c6ada882970e30, State: Initialized, Role: FOLLOWER
I20260928 02:05:25.842268 10573 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c1be5dcb0542fd81c6ada882970e30" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33023 } }
I20260928 02:05:25.842494 10573 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:05:25.842598 10573 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:05:25.842783 10573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:05:25.846582 10573 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c1be5dcb0542fd81c6ada882970e30" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33023 } }
I20260928 02:05:25.847054 10573 leader_election.cc:302] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0c1be5dcb0542fd81c6ada882970e30; no voters:
I20260928 02:05:25.847718 10573 leader_election.cc:288] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:05:25.847846 10577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:05:25.850010 10573 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:05:25.854933 10577 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [term 2 LEADER]: Becoming Leader. State: Replica: d0c1be5dcb0542fd81c6ada882970e30, State: Running, Role: LEADER
I20260928 02:05:25.855623 10577 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c1be5dcb0542fd81c6ada882970e30" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33023 } }
I20260928 02:05:25.858361 10578 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d0c1be5dcb0542fd81c6ada882970e30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c1be5dcb0542fd81c6ada882970e30" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33023 } } }
I20260928 02:05:25.858774 10578 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:25.859364 10579 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0c1be5dcb0542fd81c6ada882970e30. Latest consensus state: current_term: 2 leader_uuid: "d0c1be5dcb0542fd81c6ada882970e30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c1be5dcb0542fd81c6ada882970e30" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33023 } } }
I20260928 02:05:25.859730 10579 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:25.863286 10592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:05:25.867710 10592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:05:25.868700 10592 catalog_manager.cc:1259] Loaded cluster ID: 12dd774fb9f949a4b35dc6072b179a3b
I20260928 02:05:25.868803 10592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:05:25.871418 10592 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:05:25.872439 10592 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d0c1be5dcb0542fd81c6ada882970e30: Loaded TSK: 0
I20260928 02:05:25.873551 10592 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:05:26.149048 10575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:26.149329 10575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:26.149488 10575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:26.157150 10575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:26.157259 10575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:26.157437 10575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:05:26.167402 10575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:45099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.129.keytab
--principal=oryx/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:40179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=41623
--tserver_master_addrs=127.1.182.190:33023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:26.168753 10575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:26.170034 10575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:26.184845 10601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:26.185101 10575 server_base.cc:1034] running on GCE node
W20260928 02:05:26.184875 10603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:26.185321 10600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:26.186003 10575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:26.186980 10575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:26.188203 10575 hybrid_clock.cc:648] HybridClock initialized: now 1790561126188155 us; error 40 us; skew 500 ppm
I20260928 02:05:26.188524 10575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:26 dist-test-slave-23bp krb5kdc[9984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561126, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:26.191928 10575 init.cc:377] Logged in from keytab as oryx/127.1.182.129@KRBTEST.COM (short username oryx)
I20260928 02:05:26.193537 10575 webserver.cc:460] Webserver started at http://127.1.182.129:41623/ using document root <none> and password file <none>
I20260928 02:05:26.194360 10575 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:26.194505 10575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:26.201543 10575 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:05:26.204860 10610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:26.206532 10575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:05:26.206707 10575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "af998492a38d4b208c4a5fac8848a60a"
format_stamp: "Formatted at 2026-09-28 02:05:23 on dist-test-slave-23bp"
server_key: "6dea08487d50a382a4b94c5522d184bc"
server_key_iv: "c38cb79497a673f0b4f004383dc3a83a"
server_key_version: "encryptionkey@0"
I20260928 02:05:26.208088 10575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:26.231662 10575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:26.236917 10575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:26.237196 10575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:26.238322 10575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:26.239907 10575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:26.240011 10575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:26.240120 10575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:26.240188 10575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:26.273221 10575 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:40179
I20260928 02:05:26.273295 10723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:40179 every 8 connection(s)
I20260928 02:05:26.274736 10575 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:05:26.283301 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 10575
I20260928 02:05:26.283902 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:40979
--local_ip_for_outbound_sockets=127.1.182.130
--tserver_master_addrs=127.1.182.190:33023
--webserver_port=42117
--webserver_interface=127.1.182.130
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:45099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.182.130
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.130.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:05:26 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:26 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:05:26.614421 10728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:26.614696 10728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:26.614845 10728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:26.622195 10728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:26.622298 10728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:26.622505 10728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:05:26.632221 10728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:45099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.130.keytab
--principal=oryx/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:40979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=42117
--tserver_master_addrs=127.1.182.190:33023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:26.633567 10728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:26.634868 10728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:26.648300 10734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:26.648379 10735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:26.649636 10737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:26.650031 10728 server_base.cc:1034] running on GCE node
I20260928 02:05:26.650696 10728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:26.651770 10728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:26.652984 10728 hybrid_clock.cc:648] HybridClock initialized: now 1790561126652923 us; error 53 us; skew 500 ppm
I20260928 02:05:26.653345 10728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:26 dist-test-slave-23bp krb5kdc[9984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561126, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:26.656936 10728 init.cc:377] Logged in from keytab as oryx/127.1.182.130@KRBTEST.COM (short username oryx)
I20260928 02:05:26.658509 10728 webserver.cc:460] Webserver started at http://127.1.182.130:42117/ using document root <none> and password file <none>
I20260928 02:05:26.659256 10728 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:26.659410 10728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:26.665107 10728 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 02:05:26.668383 10744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:26.670007 10728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:05:26.670174 10728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "23cb00a9a39a4bb89a4cdeba777a11bb"
format_stamp: "Formatted at 2026-09-28 02:05:24 on dist-test-slave-23bp"
server_key: "9f3825b96eba0c0866a502e4f2df5778"
server_key_iv: "168b20e32b6530e44c6141755f63e384"
server_key_version: "encryptionkey@0"
I20260928 02:05:26.671147 10728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:26.695358 10728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:26.699807 10728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:26.700131 10728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:26.701150 10728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:26.702589 10728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:26.702719 10728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:26.702853 10728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:26.702948 10728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:26.741943 10728 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:40979
I20260928 02:05:26.741989 10857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:40979 every 8 connection(s)
I20260928 02:05:26.743595 10728 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:05:26.745867 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 10728
I20260928 02:05:26.746491 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:44457
--local_ip_for_outbound_sockets=127.1.182.131
--tserver_master_addrs=127.1.182.190:33023
--webserver_port=43567
--webserver_interface=127.1.182.131
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:45099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.182.131
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.131.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:05:26 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:26 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:05:27.369206 10862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:27.369583 10862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:27.369808 10862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:27.381481 10862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:27.381620 10862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:27.381862 10862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:05:27.398263 10862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:45099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/krb5kdc/oryx_127.1.182.131.keytab
--principal=oryx/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:44457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=43567
--tserver_master_addrs=127.1.182.190:33023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:27.400192 10862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:27.401997 10862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:27.423569 10869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:27.424647 10868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:27.430729 10862 server_base.cc:1034] running on GCE node
W20260928 02:05:27.431011 10871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:27.431790 10862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:27.432973 10862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:27.434510 10862 hybrid_clock.cc:648] HybridClock initialized: now 1790561127434433 us; error 89 us; skew 500 ppm
I20260928 02:05:27.434902 10862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:27 dist-test-slave-23bp krb5kdc[9984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561127, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:27.439550 10862 init.cc:377] Logged in from keytab as oryx/127.1.182.131@KRBTEST.COM (short username oryx)
I20260928 02:05:27.441725 10862 webserver.cc:460] Webserver started at http://127.1.182.131:43567/ using document root <none> and password file <none>
I20260928 02:05:27.442648 10862 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:27.442811 10862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260928 02:05:27.448467 10727 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.160s user 0.005s sys 0.007s
I20260928 02:05:27.449613 10862 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
W20260928 02:05:27.450940 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:26.277784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:37415 (local address 127.1.182.190:33023)
0928 02:05:26.278627 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:26.278644 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:26.278853 (+ 209us) server_negotiation.cc:407] Connection header received
0928 02:05:26.281011 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:26.281024 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:26.281354 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:26.281722 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:26.283194 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:26.284590 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:26.286075 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:26.286712 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:26.286922 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:27.450035 (+1163113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.129:37415: BlockingRecv error: recv got EOF from 127.1.182.129:37415 (error 108)
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":277,"threads_started":1}
I20260928 02:05:27.451766 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:26.277892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.129:37415)
0928 02:05:26.278680 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:26.278729 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:26.280465 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:26.281866 (+ 1401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:26.281879 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:26.282472 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:26.282986 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:26.283003 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:26.284754 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:26.284758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:26.285934 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:26.285946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:26.286946 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:26.286953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:26.287255 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:26.287926 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:26.287947 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:27.449636 (+1161689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":333,"threads_started":1}
W20260928 02:05:27.452903 10724 heartbeater.cc:644] Failed to heartbeat to 127.1.182.190:33023 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.182.190:33023: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
I20260928 02:05:27.458993 10878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:27.460784 10862 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.004s
I20260928 02:05:27.460999 10862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "23d534de6bd84592bcc63ced298c73e1"
format_stamp: "Formatted at 2026-09-28 02:05:24 on dist-test-slave-23bp"
server_key: "21d90bc7b238d59dcd1a9a3999600718"
server_key_iv: "0031f5cf9a5aca3d611b2a80b1178bc3"
server_key_version: "encryptionkey@0"
I20260928 02:05:27.462237 10862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 28 02:05:27 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:27 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:05:27.468757 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:27.454316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:39601 (local address 127.1.182.190:33023)
0928 02:05:27.454526 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:27.454535 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:27.454563 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:05:27.454779 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:27.454787 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:27.454900 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:27.455032 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:27.456295 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:27.457805 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:27.460546 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:27.461179 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:27.461310 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:27.468468 (+ 7158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.129:39601: BlockingRecv error: recv got EOF from 127.1.182.129:39601 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:05:27.469517 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:27.453877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.129:39601)
0928 02:05:27.454145 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:27.454442 (+ 297us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:27.454613 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:27.455118 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:27.455126 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:27.455793 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:27.456098 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:27.456114 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:27.457983 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:27.457992 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:27.460353 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:27.460375 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:27.461423 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:27.461432 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:27.461705 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:27.462557 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:27.462591 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:27.469253 (+ 6662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78}
Sep 28 02:05:27 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:27 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:27.484447 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:27.470463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.129:53777)
0928 02:05:27.470789 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:27.470811 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:27.470970 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:27.471417 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:27.471425 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:27.471997 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:27.472280 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:27.472307 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:27.473676 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:27.473686 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:27.476285 (+ 2599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:27.476308 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:27.477358 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:27.477368 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:27.477645 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:27.478463 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:27.478498 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:27.484033 (+ 5535us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181}
W20260928 02:05:27.484869 10724 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:05:27.485160 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:27.470362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:53777 (local address 127.1.182.190:33023)
0928 02:05:27.470499 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:27.470505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:27.470905 (+ 400us) server_negotiation.cc:407] Connection header received
0928 02:05:27.471151 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:27.471156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:27.471231 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:27.471332 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:27.472472 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:27.473468 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:27.476475 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:27.477153 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:27.477284 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:27.484249 (+ 6965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.129:53777: BlockingRecv error: recv got EOF from 127.1.182.129:53777 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:05:27.487854 10862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:27.492530 10862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:27.492836 10862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:27.493985 10862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:27.495558 10862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:27.495647 10862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:27.495814 10862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:27.495947 10862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:27.541862 10862 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:44457
I20260928 02:05:27.542035 10991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:44457 every 8 connection(s)
I20260928 02:05:27.543536 10862 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:05:27.549928 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 10862
Sep 28 02:05:27 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:27 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:27.585534 10996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:27.551210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45234 (local address 127.1.182.190:33023)
0928 02:05:27.563165 (+ 11955us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:27.563169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:27.563183 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:05:27.563221 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:27.563224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:27.563274 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:27.563388 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:27.564879 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:27.565655 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:27.567220 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:27.568249 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:27.568435 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:27.574218 (+ 5783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:27.574235 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:27.574271 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:27.574346 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:27.580091 (+ 5745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:27.581069 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:27.581085 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:27.581101 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:27.581404 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:27.582211 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:27.582220 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:27.582226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:27.583194 (+ 968us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:27.583645 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:27.584321 (+ 676us) server_negotiation.cc:299] Negotiation successful
0928 02:05:27.584889 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11787,"thread_start_us":11681,"threads_started":1}
W20260928 02:05:27.990819 10860 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.228s user 0.007s sys 0.018s
W20260928 02:05:27.992496 10861 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:26.748606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:53359 (local address 127.1.182.190:33023)
0928 02:05:26.752310 (+ 3704us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:26.752319 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:26.752340 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:05:26.752405 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:26.752412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:26.752511 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:26.752624 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:26.755625 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:26.756853 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:26.760197 (+ 3344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:26.760919 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:26.761074 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:27.992236 (+1231162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.130:53359: BlockingRecv error: recv got EOF from 127.1.182.130:53359 (error 108)
Metrics: {"server-negotiator.queue_time_us":3560,"thread_start_us":3442,"threads_started":1}
I20260928 02:05:27.993785 10860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:26.747784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.130:53359)
0928 02:05:26.748786 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:26.748842 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:26.751624 (+ 2782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:26.753067 (+ 1443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:26.753086 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:26.754220 (+ 1134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:26.755389 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:26.755424 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:26.757044 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:26.757052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:26.760005 (+ 2953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:26.760027 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:26.761220 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:26.761230 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:26.761658 (+ 428us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:26.762710 (+ 1052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:26.762754 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:27.992133 (+1229379us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":860,"threads_started":1}
W20260928 02:05:27.994738 10858 heartbeater.cc:644] Failed to heartbeat to 127.1.182.190:33023 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.182.190:33023: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Sep 28 02:05:27 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:27 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:28.007797 10860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:27.995379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.130:37413)
0928 02:05:27.995697 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:27.995715 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:27.995865 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:27.996416 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:27.996425 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:27.997079 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:27.997381 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:27.997398 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:27.998482 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:27.998490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:28.000870 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:28.000891 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:28.001661 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:28.001671 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:28.001941 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:28.002718 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:28.002752 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:28.007497 (+ 4745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161}
W20260928 02:05:28.007922 10861 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:27.995525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:37413 (local address 127.1.182.190:33023)
0928 02:05:27.995690 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:27.995695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:27.995795 (+ 100us) server_negotiation.cc:407] Connection header received
0928 02:05:27.996003 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:27.996008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:27.996094 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:27.996190 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:27.997574 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:27.998299 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:28.001047 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:28.001445 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:28.001540 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:28.007681 (+ 6141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.130:37413: BlockingRecv error: recv got EOF from 127.1.182.130:37413 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 02:05:28 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:28 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:28.019322 10860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:28.008761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.130:40543)
0928 02:05:28.008921 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:28.008938 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:28.009028 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:28.009504 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:28.009508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:28.009933 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:28.010094 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:28.010102 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:28.012205 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:28.012209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:28.013245 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:28.013256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:28.014604 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:28.014609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:28.014746 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:28.015406 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:28.015423 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:28.019119 (+ 3696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 02:05:28.019649 10858 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:05:28.019855 10861 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:28.009070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:40543 (local address 127.1.182.190:33023)
0928 02:05:28.009215 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:28.009219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:28.009232 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:05:28.009267 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:28.009271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:28.009329 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:28.009476 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:28.010280 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:28.012043 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:28.013385 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:28.014436 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:28.014601 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:28.019464 (+ 4863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.130:40543: BlockingRecv error: recv got EOF from 127.1.182.130:40543 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 02:05:28 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:28 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:28.495699 10998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:28.485404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.129:51471)
0928 02:05:28.486062 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:28.486086 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:28.486225 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:28.486765 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:28.486769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:28.487378 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:28.487569 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:28.487578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:28.488727 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:28.488732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:28.489677 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:28.489688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:28.490399 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:28.490405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:28.490581 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:28.491207 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:28.491224 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:28.495340 (+ 4116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":408,"threads_started":1}
W20260928 02:05:28.495764 10861 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:28.485580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:51471 (local address 127.1.182.190:33023)
0928 02:05:28.485868 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:28.485879 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:28.486190 (+ 311us) server_negotiation.cc:407] Connection header received
0928 02:05:28.486380 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:28.486389 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:28.486559 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:28.486738 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:28.487703 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:28.488600 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:28.489816 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:28.490218 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:28.490343 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:28.495546 (+ 5203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.129:51471: BlockingRecv error: recv got EOF from 127.1.182.129:51471 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 02:05:28.683696 10994 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.117s user 0.005s sys 0.009s
W20260928 02:05:28.685233 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:27.547770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:51965 (local address 127.1.182.190:33023)
0928 02:05:27.548005 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:27.548011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:27.549209 (+ 1198us) server_negotiation.cc:407] Connection header received
0928 02:05:27.551503 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:27.551513 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:27.551622 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:27.551709 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:27.554536 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:27.556066 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:27.559236 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:27.560165 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:27.560289 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:28.684963 (+1124674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.131:51965: BlockingRecv error: recv got EOF from 127.1.182.131:51965 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:05:28.685978 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:27.548185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.131:51965)
0928 02:05:27.549035 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:27.549082 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:27.551147 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:27.552361 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:27.552378 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:27.553291 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:27.554288 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:27.554318 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:27.556342 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:27.556350 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:27.559014 (+ 2664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:27.559033 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:27.560733 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:27.560742 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:27.561150 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:27.567036 (+ 5886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:27.567123 (+ 87us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:28.684772 (+1117649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":381,"threads_started":1}
W20260928 02:05:28.686875 10992 heartbeater.cc:644] Failed to heartbeat to 127.1.182.190:33023 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.182.190:33023: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Sep 28 02:05:28 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:28 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:28.696166 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:28.687633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.131:60509)
0928 02:05:28.687837 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:28.687864 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:28.687959 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:28.688235 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:28.688238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:28.688701 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:28.688871 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:28.688880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:28.689822 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:28.689826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:28.690884 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:28.690895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:28.691643 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:28.691648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:28.691769 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:28.692344 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:28.692368 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:28.695875 (+ 3507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 02:05:28.696316 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:28.687744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:60509 (local address 127.1.182.190:33023)
0928 02:05:28.687945 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:28.687950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:28.687965 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:05:28.688058 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:28.688062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:28.688117 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:28.688195 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:28.688990 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:28.689699 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:28.691006 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:28.691497 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:28.691589 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:28.696072 (+ 4483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.131:60509: BlockingRecv error: recv got EOF from 127.1.182.131:60509 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 28 02:05:28 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:28 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:28.703862 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:28.696769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.131:34145)
0928 02:05:28.696925 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:28.696948 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:28.697019 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:28.697251 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:28.697253 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:28.697512 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:28.697648 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:28.697654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:28.698539 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:28.698543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:28.699500 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:28.699510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:28.700117 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:28.700120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:28.700233 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:28.700714 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:28.700729 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:28.703691 (+ 2962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 02:05:28.704106 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:28.696859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:34145 (local address 127.1.182.190:33023)
0928 02:05:28.696996 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:28.697000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:28.697013 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:05:28.697089 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:28.697092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:28.697139 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:28.697210 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:28.697754 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:28.698435 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:28.699616 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:28.700001 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:28.700089 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:28.703860 (+ 3771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.131:34145: BlockingRecv error: recv got EOF from 127.1.182.131:34145 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 02:05:28.704166 10992 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 02:05:29 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:29 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:29.028761 10999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:29.020208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.130:54049)
0928 02:05:29.020733 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:29.020749 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:29.020862 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:29.021161 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:29.021164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:29.021560 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:29.021698 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:29.021705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:29.022620 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:29.022624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:29.023582 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:29.023592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:29.024268 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:29.024275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:29.024459 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:29.025067 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:29.025084 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:29.028537 (+ 3453us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":278,"threads_started":1}
W20260928 02:05:29.028939 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:29.020270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:54049 (local address 127.1.182.190:33023)
0928 02:05:29.020507 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:29.020513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:29.020828 (+ 315us) server_negotiation.cc:407] Connection header received
0928 02:05:29.020977 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:29.020981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:29.021042 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:29.021113 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:29.021815 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:29.022506 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:29.023704 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:29.024082 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:29.024180 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:29.028721 (+ 4541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.130:54049: BlockingRecv error: recv got EOF from 127.1.182.130:54049 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 02:05:29 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:29 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:29.506361 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:29.496649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.129:38281)
0928 02:05:29.497179 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:29.497195 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:29.497311 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:29.497673 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:29.497677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:29.498126 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:29.498327 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:29.498339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:29.499639 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:29.499643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:29.500971 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:29.500986 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:29.501699 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:29.501704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:29.501858 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:29.502466 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:29.502484 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:29.506104 (+ 3620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":316,"threads_started":1}
W20260928 02:05:29.506523 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:29.496753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:38281 (local address 127.1.182.190:33023)
0928 02:05:29.496945 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:29.496949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:29.497271 (+ 322us) server_negotiation.cc:407] Connection header received
0928 02:05:29.497436 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:29.497440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:29.497499 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:29.497582 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:29.498493 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:29.499507 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:29.501102 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:29.501538 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:29.501634 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:29.506282 (+ 4648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.129:38281: BlockingRecv error: recv got EOF from 127.1.182.129:38281 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 02:05:29 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:29 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:29.713186 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:29.704747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.131:53645)
0928 02:05:29.705384 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:29.705399 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:29.705514 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:29.705786 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:29.705789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:29.706205 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:29.706342 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:29.706350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:29.707331 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:29.707335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:29.708266 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:29.708276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:29.708897 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:29.708901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:29.709015 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:29.709540 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:29.709562 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:29.712933 (+ 3371us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":416,"threads_started":1}
W20260928 02:05:29.713433 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:29.704805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:53645 (local address 127.1.182.190:33023)
0928 02:05:29.704960 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:29.704965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:29.705486 (+ 521us) server_negotiation.cc:407] Connection header received
0928 02:05:29.705611 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:29.705614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:29.705675 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:29.705752 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:29.706461 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:29.707215 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:29.708379 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:29.708756 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:29.708844 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:29.713105 (+ 4261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.131:53645: BlockingRecv error: recv got EOF from 127.1.182.131:53645 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 02:05:30 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:30 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:30.038142 11002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:30.029658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.130:48231)
0928 02:05:30.030092 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:30.030107 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:30.030218 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:30.030543 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:30.030548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:30.030982 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:30.031223 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:30.031232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:30.032216 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:30.032219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:30.033129 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:30.033138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:30.033750 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:30.033754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:30.033876 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:30.034426 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:30.034442 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:30.037784 (+ 3342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":246,"threads_started":1}
W20260928 02:05:30.038221 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:30.029685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:48231 (local address 127.1.182.190:33023)
0928 02:05:30.029900 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:30.029909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:30.030184 (+ 275us) server_negotiation.cc:407] Connection header received
0928 02:05:30.030300 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:30.030303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:30.030393 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:30.030482 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:30.031346 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:30.032096 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:30.033241 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:30.033622 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:30.033713 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:30.038003 (+ 4290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.130:48231: BlockingRecv error: recv got EOF from 127.1.182.130:48231 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 02:05:30 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:30 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:30.516460 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:30.507365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.129:36127)
0928 02:05:30.508269 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:30.508288 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:30.508452 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:30.508802 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:30.508805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:30.509156 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:30.509317 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:30.509325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:30.510304 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:30.510308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:30.511332 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:30.511344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:30.511997 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:30.512001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:30.512116 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:30.512742 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:30.512759 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:30.516222 (+ 3463us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":637,"threads_started":1}
W20260928 02:05:30.516641 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:30.507644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:36127 (local address 127.1.182.190:33023)
0928 02:05:30.507843 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:30.507847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:30.508384 (+ 537us) server_negotiation.cc:407] Connection header received
0928 02:05:30.508586 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:30.508592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:30.508675 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:30.508754 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:30.509442 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:30.510167 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:30.511455 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:30.511873 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:30.511974 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:30.516406 (+ 4432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.129:36127: BlockingRecv error: recv got EOF from 127.1.182.129:36127 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 02:05:30 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:30 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:30.725082 11004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:30.714278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.131:55493)
0928 02:05:30.714832 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:30.714856 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:30.715005 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:30.715550 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:30.715558 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:30.716180 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:30.716406 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:30.716416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:30.717985 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:30.717991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:30.719034 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:30.719046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:30.719890 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:30.719897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:30.720102 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:30.720808 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:30.720827 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:30.724701 (+ 3874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":277,"threads_started":1}
W20260928 02:05:30.725203 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:30.714334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:55493 (local address 127.1.182.190:33023)
0928 02:05:30.714553 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:30.714561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:30.714933 (+ 372us) server_negotiation.cc:407] Connection header received
0928 02:05:30.715194 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:30.715200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:30.715308 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:30.715437 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:30.716570 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:30.717827 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:30.719219 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:30.719717 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:30.719875 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:30.724915 (+ 5040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.131:55493: BlockingRecv error: recv got EOF from 127.1.182.131:55493 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 02:05:31 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:31 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:31.047698 11005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:31.039123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.130:58537)
0928 02:05:31.039627 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:31.039646 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:31.039760 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:31.040055 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:31.040058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:31.040423 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:31.040586 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:31.040595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:31.041579 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:31.041583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:31.042482 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:31.042491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:31.043238 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:31.043242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:31.043359 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:31.043908 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:31.043924 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:31.047393 (+ 3469us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":267,"threads_started":1}
W20260928 02:05:31.048075 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:31.039172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:58537 (local address 127.1.182.190:33023)
0928 02:05:31.039418 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:31.039427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:31.039760 (+ 333us) server_negotiation.cc:407] Connection header received
0928 02:05:31.039865 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:31.039869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:31.039931 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:31.040035 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:31.040717 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:31.041458 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:31.042607 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:31.043037 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:31.043206 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:31.047633 (+ 4427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.130:58537: BlockingRecv error: recv got EOF from 127.1.182.130:58537 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 02:05:31 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:31 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:31.526149 11006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:31.517483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.129:47975)
0928 02:05:31.517950 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:31.517967 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:31.518072 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:31.518366 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:31.518369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:31.518732 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:31.518889 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:31.518897 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:31.519910 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:31.519913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:31.520890 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:31.520900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:31.521575 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:31.521579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:31.521698 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:31.522253 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:31.522269 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:31.525844 (+ 3575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":255,"threads_started":1}
W20260928 02:05:31.526293 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:31.517516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:47975 (local address 127.1.182.190:33023)
0928 02:05:31.517727 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:31.517737 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:31.518047 (+ 310us) server_negotiation.cc:407] Connection header received
0928 02:05:31.518169 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:31.518172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:31.518233 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:31.518322 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:31.519013 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:31.519800 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:31.521014 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:31.521422 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:31.521556 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:31.526073 (+ 4517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.129:47975: BlockingRecv error: recv got EOF from 127.1.182.129:47975 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 02:05:31 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:31 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:31.734757 11007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:31.726257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.131:40893)
0928 02:05:31.726704 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:31.726720 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:31.726848 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:31.727136 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:31.727139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:31.727529 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:31.727684 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:31.727693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:31.728681 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:31.728685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:31.729599 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:31.729610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:31.730339 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:31.730344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:31.730472 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:31.731264 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:31.731336 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:31.734531 (+ 3195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":236,"threads_started":1}
W20260928 02:05:31.734982 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:31.726443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:40893 (local address 127.1.182.190:33023)
0928 02:05:31.726667 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:31.726674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:31.726802 (+ 128us) server_negotiation.cc:407] Connection header received
0928 02:05:31.726939 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:31.726942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:31.727003 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:31.727101 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:31.727811 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:31.728551 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:31.729732 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:31.730163 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:31.730256 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:31.734719 (+ 4463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.131:40893: BlockingRecv error: recv got EOF from 127.1.182.131:40893 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 28 02:05:32 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:32 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:32.057638 11008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:32.048622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.130:41845)
0928 02:05:32.049153 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:32.049169 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:32.049278 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:32.049562 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:32.049565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:32.049935 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:32.050144 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:32.050154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:32.051286 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:32.051290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:32.052288 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:32.052300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:32.053003 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:32.053007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:32.053129 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:32.053685 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:32.053702 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:32.057385 (+ 3683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":296,"threads_started":1}
W20260928 02:05:32.057835 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:32.048684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:41845 (local address 127.1.182.190:33023)
0928 02:05:32.048864 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:32.048868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:32.049242 (+ 374us) server_negotiation.cc:407] Connection header received
0928 02:05:32.049374 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:32.049378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:32.049438 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:32.049522 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:32.050303 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:32.051161 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:32.052410 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:32.052882 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:32.052978 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:32.057573 (+ 4595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.130:41845: BlockingRecv error: recv got EOF from 127.1.182.130:41845 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 02:05:32 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:32 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:32.539628 11009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:32.527231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.129:39803)
0928 02:05:32.527802 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:32.527819 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:32.527932 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:32.528240 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:32.528243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:32.528652 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:32.528797 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:32.528805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:32.529826 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:32.529830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:32.530887 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:32.530900 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:32.531685 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:32.531689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:32.531809 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:32.535358 (+ 3549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:32.535375 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:32.539273 (+ 3898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":339,"threads_started":1}
W20260928 02:05:32.539779 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:32.527271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:39803 (local address 127.1.182.190:33023)
0928 02:05:32.527437 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:32.527441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:32.527899 (+ 458us) server_negotiation.cc:407] Connection header received
0928 02:05:32.528029 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:32.528035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:32.528099 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:32.528179 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:32.528920 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:32.529698 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:32.531018 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:32.531558 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:32.531655 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:32.539519 (+ 7864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.129:39803: BlockingRecv error: recv got EOF from 127.1.182.129:39803 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 02:05:32 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:32 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:32.745640 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:32.735759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.131:42743)
0928 02:05:32.736289 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:32.736304 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:32.736473 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:32.736825 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:32.736829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:32.737224 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:32.737425 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:32.737433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:32.738461 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:32.738467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:32.739480 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:32.739491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:32.740212 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:32.740218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:32.740401 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:32.741008 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:32.741024 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:32.745256 (+ 4232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":276,"threads_started":1}
W20260928 02:05:32.745724 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:32.735860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:42743 (local address 127.1.182.190:33023)
0928 02:05:32.736093 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:32.736102 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:32.736394 (+ 292us) server_negotiation.cc:407] Connection header received
0928 02:05:32.736598 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:32.736603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:32.736685 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:32.736801 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:32.737568 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:32.738316 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:32.739630 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:32.740058 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:32.740189 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:32.745448 (+ 5259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.131:42743: BlockingRecv error: recv got EOF from 127.1.182.131:42743 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 28 02:05:33 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:33 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:33.067974 11011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:33.058612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.130:34775)
0928 02:05:33.059263 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:33.059278 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:33.059433 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:33.059771 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:33.059775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:33.060201 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:33.060342 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:33.060350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:33.061381 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:33.061385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:33.062845 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:33.062856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:33.063616 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:33.063621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:33.063769 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:33.064343 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:33.064361 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:33.067745 (+ 3384us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":382,"threads_started":1}
W20260928 02:05:33.068197 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:33.058947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:34775 (local address 127.1.182.190:33023)
0928 02:05:33.059198 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:33.059204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:33.059357 (+ 153us) server_negotiation.cc:407] Connection header received
0928 02:05:33.059567 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:33.059581 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:33.059639 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:33.059741 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:33.060481 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:33.061239 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:33.062969 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:33.063457 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:33.063555 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:33.067928 (+ 4373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.130:34775: BlockingRecv error: recv got EOF from 127.1.182.130:34775 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 02:05:33 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:33 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:33.550676 11012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:33.540670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.129:44141)
0928 02:05:33.541315 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:33.541335 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:33.541467 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:33.541821 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:33.541825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:33.542352 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:33.542531 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:33.542540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:33.543980 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:33.543987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:33.545423 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:33.545436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:33.546292 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:33.546298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:33.546432 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:33.547035 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:33.547052 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:33.550444 (+ 3392us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":363,"threads_started":1}
W20260928 02:05:33.551062 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:33.540747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:44141 (local address 127.1.182.190:33023)
0928 02:05:33.540947 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:33.540951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:33.541458 (+ 507us) server_negotiation.cc:407] Connection header received
0928 02:05:33.541583 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:33.541588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:33.541667 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:33.541779 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:33.542681 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:33.543808 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:33.545571 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:33.546062 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:33.546176 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:33.550643 (+ 4467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.129:44141: BlockingRecv error: recv got EOF from 127.1.182.129:44141 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 02:05:33 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:33 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:33.755981 11013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:33.746755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.131:54011)
0928 02:05:33.747328 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:33.747344 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:33.747500 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:33.747847 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:33.747852 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:33.748319 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:33.748469 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:33.748478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:33.749501 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:33.749506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:33.750573 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:33.750585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:33.751410 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:33.751415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:33.751596 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:33.752152 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:33.752168 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:33.755656 (+ 3488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":313,"threads_started":1}
W20260928 02:05:33.756078 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:33.746943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:54011 (local address 127.1.182.190:33023)
0928 02:05:33.747172 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:33.747178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:33.747433 (+ 255us) server_negotiation.cc:407] Connection header received
0928 02:05:33.747633 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:33.747645 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:33.747710 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:33.747819 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:33.748612 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:33.749357 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:33.750710 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:33.751222 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:33.751403 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:33.755836 (+ 4433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.131:54011: BlockingRecv error: recv got EOF from 127.1.182.131:54011 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 02:05:34 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:34 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:34.079170 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:34.069006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.130:37223)
0928 02:05:34.069905 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:34.069930 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:34.070118 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:34.070513 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:34.070520 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:34.070965 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:34.071158 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:34.071168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:34.072305 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:34.072317 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:34.073317 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:34.073328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:34.074206 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:34.074213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:34.074411 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:34.075144 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:34.075167 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:34.078811 (+ 3644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":684,"threads_started":1}
W20260928 02:05:34.079258 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:34.069071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:37223 (local address 127.1.182.190:33023)
0928 02:05:34.069496 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:34.069506 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:34.070035 (+ 529us) server_negotiation.cc:407] Connection header received
0928 02:05:34.070259 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:34.070265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:34.070341 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:34.070452 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:34.071331 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:34.072136 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:34.073481 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:34.074023 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:34.074189 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:34.078989 (+ 4800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.130:37223: BlockingRecv error: recv got EOF from 127.1.182.130:37223 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 28 02:05:34 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:34 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:34.561084 11015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:34.551758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.129:37339)
0928 02:05:34.552372 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:34.552390 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:34.552510 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:34.552818 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:34.552821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:34.553292 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:34.553474 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:34.553483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:34.554622 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:34.554626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:34.555657 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:34.555670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:34.556367 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:34.556372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:34.556487 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:34.557055 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:34.557071 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:34.560730 (+ 3659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":302,"threads_started":1}
W20260928 02:05:34.561200 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:34.551917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:37339 (local address 127.1.182.190:33023)
0928 02:05:34.552088 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:34.552092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:34.552481 (+ 389us) server_negotiation.cc:407] Connection header received
0928 02:05:34.552616 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:34.552619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:34.552683 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:34.552768 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:34.553606 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:34.554487 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:34.555791 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:34.556219 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:34.556316 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:34.560960 (+ 4644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.129:37339: BlockingRecv error: recv got EOF from 127.1.182.129:37339 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 02:05:34 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:34 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:34.765625 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:34.757017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.131:34097)
0928 02:05:34.757470 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:34.757487 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:34.757600 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:34.757942 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:34.757946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:34.758354 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:34.758528 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:34.758537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:34.759546 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:34.759550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:34.760466 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:34.760477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:34.761141 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:34.761145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:34.761316 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:34.761913 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:34.761930 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:34.765395 (+ 3465us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":253,"threads_started":1}
W20260928 02:05:34.765933 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:34.757095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:34097 (local address 127.1.182.190:33023)
0928 02:05:34.757317 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:34.757323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:34.757572 (+ 249us) server_negotiation.cc:407] Connection header received
0928 02:05:34.757699 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:34.757702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:34.757764 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:34.757861 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:34.758664 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:34.759425 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:34.760594 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:34.761001 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:34.761094 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:34.765606 (+ 4512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.131:34097: BlockingRecv error: recv got EOF from 127.1.182.131:34097 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 02:05:35 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:35 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:35.090090 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:35.080099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.130:55695)
0928 02:05:35.080748 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:35.080764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:35.080884 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:35.081217 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:35.081220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:35.081643 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:35.081811 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:35.081819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:35.082861 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:35.082864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:35.083862 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:35.083873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:35.084616 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:35.084621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:35.084731 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:35.085371 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:35.085387 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:35.089799 (+ 4412us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":349,"threads_started":1}
W20260928 02:05:35.090176 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:35.080194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:55695 (local address 127.1.182.190:33023)
0928 02:05:35.080424 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:35.080429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:35.080852 (+ 423us) server_negotiation.cc:407] Connection header received
0928 02:05:35.081009 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:35.081013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:35.081079 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:35.081163 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:35.081945 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:35.082729 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:35.084015 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:35.084477 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:35.084584 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:35.089995 (+ 5411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.130:55695: BlockingRecv error: recv got EOF from 127.1.182.130:55695 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 02:05:35 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:35 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:35.571492 11018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:35.562153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.129:37717)
0928 02:05:35.562732 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:35.562750 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:35.562877 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:35.563224 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:35.563228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:35.563670 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:35.563842 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:35.563852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:35.565022 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:35.565026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:35.565996 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:35.566006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:35.566884 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:35.566889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:35.567000 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:35.567734 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:35.567751 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:35.571184 (+ 3433us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":309,"threads_started":1}
W20260928 02:05:35.571767 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:35.562187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:37717 (local address 127.1.182.190:33023)
0928 02:05:35.562408 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:35.562413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:35.562832 (+ 419us) server_negotiation.cc:407] Connection header received
0928 02:05:35.562976 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:35.562979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:35.563039 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:35.563140 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:35.564011 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:35.564875 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:35.566184 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:35.566723 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:35.566831 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:35.571480 (+ 4649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.129:37717: BlockingRecv error: recv got EOF from 127.1.182.129:37717 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 02:05:35 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:35 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:35.775707 11019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:35.766700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.131:34779)
0928 02:05:35.767324 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:35.767340 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:35.767461 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:35.767814 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:35.767817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:35.768213 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:35.768434 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:35.768442 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:35.769423 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:35.769427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:35.770384 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:35.770395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:35.771149 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:35.771154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:35.771260 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:35.771960 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:35.771977 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:35.775460 (+ 3483us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":299,"threads_started":1}
W20260928 02:05:35.775841 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:35.766803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:34779 (local address 127.1.182.190:33023)
0928 02:05:35.767059 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:35.767085 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:35.767419 (+ 334us) server_negotiation.cc:407] Connection header received
0928 02:05:35.767613 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:35.767617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:35.767680 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:35.767785 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:35.768560 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:35.769289 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:35.770525 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:35.770951 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:35.771049 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:35.775630 (+ 4581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.131:34779: BlockingRecv error: recv got EOF from 127.1.182.131:34779 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 02:05:36 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:36 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:36.100751 11020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:36.091264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.130:39069)
0928 02:05:36.091770 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:36.091787 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:36.091918 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:36.092213 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:36.092216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:36.092613 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:36.092769 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:36.092778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:36.093961 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:36.093965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:36.095211 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:36.095227 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:36.096105 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:36.096113 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:36.096288 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:36.096976 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:36.096993 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:36.100410 (+ 3417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":263,"threads_started":1}
W20260928 02:05:36.100852 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:36.091435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:39069 (local address 127.1.182.190:33023)
0928 02:05:36.091697 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:36.091705 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:36.091864 (+ 159us) server_negotiation.cc:407] Connection header received
0928 02:05:36.092022 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:36.092026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:36.092091 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:36.092171 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:36.092919 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:36.093822 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:36.095383 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:36.095902 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:36.096044 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:36.100602 (+ 4558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.130:39069: BlockingRecv error: recv got EOF from 127.1.182.130:39069 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 28 02:05:36 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:36 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:36.581859 11021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:36.572514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.129:46569)
0928 02:05:36.573039 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:36.573055 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:36.573214 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:36.573570 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:36.573577 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:36.574042 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:36.574228 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:36.574237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:36.575447 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:36.575453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:36.576401 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:36.576412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:36.577112 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:36.577118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:36.577298 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:36.577884 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:36.577900 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:36.581499 (+ 3599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":283,"threads_started":1}
W20260928 02:05:36.581983 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:36.572562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:46569 (local address 127.1.182.190:33023)
0928 02:05:36.572848 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:36.572857 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:36.573148 (+ 291us) server_negotiation.cc:407] Connection header received
0928 02:05:36.573360 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:36.573364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:36.573423 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:36.573529 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:36.574370 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:36.575265 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:36.576542 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:36.576958 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:36.577096 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:36.581731 (+ 4635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.129:46569: BlockingRecv error: recv got EOF from 127.1.182.129:46569 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Sep 28 02:05:36 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:36 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:36.785377 11022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:36.776629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.131:57523)
0928 02:05:36.777116 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:36.777131 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:36.777242 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:36.777618 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:36.777622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:36.778023 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:36.778201 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:36.778210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:36.779299 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:36.779304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:36.780237 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:36.780248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:36.781022 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:36.781026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:36.781136 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:36.781802 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:36.781819 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:36.785128 (+ 3309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":259,"threads_started":1}
W20260928 02:05:36.785560 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:36.776673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:57523 (local address 127.1.182.190:33023)
0928 02:05:36.776921 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:36.776929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:36.777221 (+ 292us) server_negotiation.cc:407] Connection header received
0928 02:05:36.777413 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:36.777418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:36.777485 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:36.777579 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:36.778371 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:36.779152 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:36.780370 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:36.780868 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:36.780968 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:36.785315 (+ 4347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.131:57523: BlockingRecv error: recv got EOF from 127.1.182.131:57523 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 28 02:05:37 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:37 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:37.111155 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:37.101794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.130:41319)
0928 02:05:37.102298 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:37.102314 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:37.102453 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:37.102899 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:37.102910 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:37.103374 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:37.103520 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:37.103528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:37.104578 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:37.104582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:37.105523 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:37.105534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:37.106312 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:37.106317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:37.106441 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:37.107107 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:37.107124 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:37.110745 (+ 3621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":258,"threads_started":1}
W20260928 02:05:37.111279 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:37.101846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:41319 (local address 127.1.182.190:33023)
0928 02:05:37.102072 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:37.102076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:37.102413 (+ 337us) server_negotiation.cc:407] Connection header received
0928 02:05:37.102607 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:37.102612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:37.102736 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:37.102850 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:37.103654 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:37.104442 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:37.105659 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:37.106120 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:37.106219 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:37.110961 (+ 4742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.130:41319: BlockingRecv error: recv got EOF from 127.1.182.130:41319 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 02:05:37 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:37 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:37.592567 11024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:37.583040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.129:41535)
0928 02:05:37.583599 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:37.583616 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:37.583725 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:37.584059 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:37.584062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:37.584559 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:37.584756 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:37.584765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:37.586106 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:37.586110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:37.587193 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:37.587205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:37.587915 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:37.587919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:37.588029 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:37.588672 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:37.588690 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:37.592318 (+ 3628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":287,"threads_started":1}
W20260928 02:05:37.592649 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:37.583316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:41535 (local address 127.1.182.190:33023)
0928 02:05:37.583489 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:37.583494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:37.583691 (+ 197us) server_negotiation.cc:407] Connection header received
0928 02:05:37.583824 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:37.583827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:37.583890 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:37.583984 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:37.584930 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:37.585935 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:37.587370 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:37.587773 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:37.587874 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:37.592457 (+ 4583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.129:41535: BlockingRecv error: recv got EOF from 127.1.182.129:41535 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 02:05:37 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:37 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:37.802327 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:37.786468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.131:43663)
0928 02:05:37.787001 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:37.787017 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:37.787204 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:37.787568 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:37.787575 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:37.787978 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:37.788121 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:37.788129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:37.789224 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:37.789228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:37.791793 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:37.791809 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:37.792782 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:37.792787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:37.792918 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:37.798458 (+ 5540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:37.798478 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:37.801996 (+ 3518us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":288,"threads_started":1}
W20260928 02:05:37.802704 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:37.786585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:43663 (local address 127.1.182.190:33023)
0928 02:05:37.786853 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:37.786859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:37.787115 (+ 256us) server_negotiation.cc:407] Connection header received
0928 02:05:37.787331 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:37.787337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:37.787414 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:37.787526 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:37.788287 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:37.789029 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:37.791923 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:37.792615 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:37.792736 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:37.802188 (+ 9452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.131:43663: BlockingRecv error: recv got EOF from 127.1.182.131:43663 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 02:05:38 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:38 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:38.121555 11026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:38.112068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.130:55557)
0928 02:05:38.112633 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:38.112649 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:38.112845 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:38.113179 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:38.113184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:38.113646 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:38.113808 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:38.113817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:38.114820 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:38.114827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:38.115858 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:38.115869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:38.116592 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:38.116597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:38.116773 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:38.117401 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:38.117418 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:38.121324 (+ 3906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":322,"threads_started":1}
W20260928 02:05:38.121716 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:38.112182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:55557 (local address 127.1.182.190:33023)
0928 02:05:38.112435 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:38.112443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:38.112739 (+ 296us) server_negotiation.cc:407] Connection header received
0928 02:05:38.112965 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:38.112971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:38.113043 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:38.113146 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:38.113972 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:38.114693 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:38.116001 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:38.116439 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:38.116584 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:38.121490 (+ 4906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.130:55557: BlockingRecv error: recv got EOF from 127.1.182.130:55557 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 02:05:38 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:38 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:38.602169 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:38.593625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.129:46847)
0928 02:05:38.594119 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:38.594135 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:38.594249 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:38.594565 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:38.594568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:38.594982 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:38.595188 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:38.595197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:38.596211 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:38.596215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:38.597118 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:38.597129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:38.597836 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:38.597841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:38.597965 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:38.598619 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:38.598636 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:38.601953 (+ 3317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":286,"threads_started":1}
W20260928 02:05:38.602424 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:38.593803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:46847 (local address 127.1.182.190:33023)
0928 02:05:38.594044 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:38.594050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:38.594217 (+ 167us) server_negotiation.cc:407] Connection header received
0928 02:05:38.594364 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:38.594368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:38.594428 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:38.594507 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:38.595360 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:38.596099 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:38.597279 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:38.597693 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:38.597802 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:38.602130 (+ 4328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.129:46847: BlockingRecv error: recv got EOF from 127.1.182.129:46847 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 28 02:05:38 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:38 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:38.812451 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:38.803408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.131:40873)
0928 02:05:38.803877 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:38.803892 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:38.804045 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:38.804377 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:38.804381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:38.804782 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:38.804943 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:38.804952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:38.806113 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:38.806120 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:38.807286 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:38.807297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:38.808044 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:38.808053 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:38.808225 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:38.808775 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:38.808792 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:38.812103 (+ 3311us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":262,"threads_started":1}
W20260928 02:05:38.812489 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:38.803464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:40873 (local address 127.1.182.190:33023)
0928 02:05:38.803693 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:38.803702 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:38.803977 (+ 275us) server_negotiation.cc:407] Connection header received
0928 02:05:38.804165 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:38.804170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:38.804245 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:38.804350 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:38.805086 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:38.805982 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:38.807416 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:38.807889 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:38.808030 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:38.812272 (+ 4242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.131:40873: BlockingRecv error: recv got EOF from 127.1.182.131:40873 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 02:05:39 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:39 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:39.131702 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:39.122611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.130:59537)
0928 02:05:39.123391 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:39.123413 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:39.123548 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:39.123842 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:39.123845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:39.124244 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:39.124427 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:39.124436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:39.125466 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:39.125470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:39.126377 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:39.126389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:39.127130 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:39.127135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:39.127250 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:39.127932 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:39.127949 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:39.131467 (+ 3518us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":555,"threads_started":1}
W20260928 02:05:39.131975 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:39.122716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:59537 (local address 127.1.182.190:33023)
0928 02:05:39.122940 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:39.122948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:39.123484 (+ 536us) server_negotiation.cc:407] Connection header received
0928 02:05:39.123633 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:39.123636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:39.123696 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:39.123778 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:39.124568 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:39.125340 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:39.126526 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:39.126966 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:39.127137 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:39.131673 (+ 4536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.130:59537: BlockingRecv error: recv got EOF from 127.1.182.130:59537 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 02:05:39 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:39 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:39.612805 11030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:39.603255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.129:38347)
0928 02:05:39.603882 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:39.603900 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:39.604019 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:39.604335 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:39.604339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:39.604798 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:39.604980 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:39.604989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:39.605972 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:39.605976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:39.607054 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:39.607122 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:39.607937 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:39.607944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:39.608138 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:39.608926 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:39.608947 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:39.612550 (+ 3603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":340,"threads_started":1}
W20260928 02:05:39.613006 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:39.603340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:38347 (local address 127.1.182.190:33023)
0928 02:05:39.603520 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:39.603524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:39.604005 (+ 481us) server_negotiation.cc:407] Connection header received
0928 02:05:39.604137 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:39.604140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:39.604204 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:39.604291 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:39.605113 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:39.605837 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:39.607261 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:39.607690 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:39.607809 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:39.612726 (+ 4917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.129:38347: BlockingRecv error: recv got EOF from 127.1.182.129:38347 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 02:05:39 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:39 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:39.822449 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:39.813467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.131:51877)
0928 02:05:39.814079 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:39.814096 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:39.814255 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:39.814601 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:39.814606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:39.815035 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:39.815268 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:39.815277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:39.816284 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:39.816290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:39.817204 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:39.817215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:39.817966 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:39.817971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:39.818142 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:39.818752 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:39.818770 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:39.822140 (+ 3370us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":391,"threads_started":1}
W20260928 02:05:39.822629 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:39.813587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:51877 (local address 127.1.182.190:33023)
0928 02:05:39.813771 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:39.813777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:39.814185 (+ 408us) server_negotiation.cc:407] Connection header received
0928 02:05:39.814381 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:39.814387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:39.814457 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:39.814567 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:39.815425 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:39.816149 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:39.817354 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:39.817807 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:39.817936 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:39.822376 (+ 4440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.131:51877: BlockingRecv error: recv got EOF from 127.1.182.131:51877 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 02:05:40 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:40 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:40.142477 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:40.132804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.130:47231)
0928 02:05:40.133418 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:40.133434 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:40.133597 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:40.133988 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:40.133992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:40.134397 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:40.134567 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:40.134575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:40.135640 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:40.135648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:40.136567 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:40.136578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:40.137345 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:40.137351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:40.137523 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:40.138171 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:40.138188 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:40.142220 (+ 4032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":314,"threads_started":1}
W20260928 02:05:40.142666 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:40.132880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:47231 (local address 127.1.182.190:33023)
0928 02:05:40.133150 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:40.133158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:40.133527 (+ 369us) server_negotiation.cc:407] Connection header received
0928 02:05:40.133734 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:40.133745 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:40.133834 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:40.133940 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:40.134712 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:40.135500 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:40.136707 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:40.137157 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:40.137294 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:40.142409 (+ 5115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.130:47231: BlockingRecv error: recv got EOF from 127.1.182.130:47231 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 28 02:05:40 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:40 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:40.623006 11033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:40.613652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.129:34219)
0928 02:05:40.614170 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:40.614186 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:40.614298 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:40.614595 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:40.614598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:40.615009 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:40.615170 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:40.615178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:40.616189 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:40.616194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:40.617106 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:40.617117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:40.617765 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:40.617769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:40.617900 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:40.618656 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:40.618672 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:40.622778 (+ 4106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":327,"threads_started":1}
W20260928 02:05:40.623217 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:40.613734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:34219 (local address 127.1.182.190:33023)
0928 02:05:40.613998 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:40.614004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:40.614269 (+ 265us) server_negotiation.cc:407] Connection header received
0928 02:05:40.614394 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:40.614397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:40.614472 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:40.614548 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:40.615346 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:40.616066 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:40.617234 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:40.617647 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:40.617744 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:40.622977 (+ 5233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.129:34219: BlockingRecv error: recv got EOF from 127.1.182.129:34219 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 28 02:05:40 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:40 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:40.832870 11034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:40.823593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.131:37373)
0928 02:05:40.824200 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:40.824221 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:40.824435 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:40.824771 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:40.824776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:40.825341 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:40.825522 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:40.825531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:40.826569 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:40.826574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:40.827604 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:40.827617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:40.828338 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:40.828344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:40.828522 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:40.829126 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:40.829142 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:40.832489 (+ 3347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":301,"threads_started":1}
W20260928 02:05:40.832926 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:40.823681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:37373 (local address 127.1.182.190:33023)
0928 02:05:40.823908 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:40.823917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:40.824356 (+ 439us) server_negotiation.cc:407] Connection header received
0928 02:05:40.824568 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:40.824578 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:40.824637 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:40.824740 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:40.825664 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:40.826431 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:40.827746 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:40.828158 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:40.828309 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:40.832695 (+ 4386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.131:37373: BlockingRecv error: recv got EOF from 127.1.182.131:37373 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 02:05:41 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:41 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:41.153198 11035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:41.143524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.130:48155)
0928 02:05:41.144030 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:41.144056 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:41.144175 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:41.144491 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:41.144494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:41.144960 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:41.145116 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:41.145125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:41.146263 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:41.146268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:41.147316 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:41.147329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:41.148050 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:41.148054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:41.148173 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:41.148851 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:41.148868 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:41.152881 (+ 4013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":298,"threads_started":1}
W20260928 02:05:41.153266 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:41.143693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:48155 (local address 127.1.182.190:33023)
0928 02:05:41.143886 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:41.143891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:41.144147 (+ 256us) server_negotiation.cc:407] Connection header received
0928 02:05:41.144274 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:41.144277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:41.144352 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:41.144434 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:41.145253 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:41.146105 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:41.147465 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:41.147927 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:41.148031 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:41.153050 (+ 5019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.130:48155: BlockingRecv error: recv got EOF from 127.1.182.130:48155 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 02:05:41 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:41 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.129@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:41.633698 11036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:41.624658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.129:42443)
0928 02:05:41.625354 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:41.625376 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:41.625568 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:41.625916 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:41.625919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:41.626264 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:41.626495 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:41.626504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:41.627721 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:41.627726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:41.628628 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:41.628639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:41.629416 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:41.629420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:41.629544 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:41.630113 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:41.630129 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:41.633307 (+ 3178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":398,"threads_started":1}
W20260928 02:05:41.633883 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:41.624733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:42443 (local address 127.1.182.190:33023)
0928 02:05:41.625090 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:41.625100 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:41.625456 (+ 356us) server_negotiation.cc:407] Connection header received
0928 02:05:41.625701 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:41.625705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:41.625789 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:41.625871 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:41.626648 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:41.627569 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:41.628783 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:41.629286 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:41.629396 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:41.633578 (+ 4182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.129:42443: BlockingRecv error: recv got EOF from 127.1.182.129:42443 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 02:05:41 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:41 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.131@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:41.843026 11037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:41.833908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.131:40871)
0928 02:05:41.834436 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:41.834453 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:41.834612 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:41.834970 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:41.834974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:41.835444 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:41.835582 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:41.835590 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:41.836634 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:41.836640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:41.837554 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:41.837565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:41.838275 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:41.838281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:41.838451 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:41.839167 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:41.839184 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:41.842618 (+ 3434us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":279,"threads_started":1}
W20260928 02:05:41.843159 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:41.833962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:40871 (local address 127.1.182.190:33023)
0928 02:05:41.834384 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:41.834391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:41.834535 (+ 144us) server_negotiation.cc:407] Connection header received
0928 02:05:41.834766 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:41.834771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:41.834833 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:41.834939 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:41.835732 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:41.836481 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:41.837702 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:41.838124 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:41.838276 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:41.842856 (+ 4580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.131:40871: BlockingRecv error: recv got EOF from 127.1.182.131:40871 (error 108)
Metrics: {"server-negotiator.queue_time_us":205}
Sep 28 02:05:42 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:05:42 dist-test-slave-23bp krb5kdc[9984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.182.130@KRBTEST.COM for oryx/127.1.182.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:05:42.163606 11038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:42.154430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33023 (local address 127.1.182.130:56237)
0928 02:05:42.154913 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:42.154929 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:42.155039 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:42.155409 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:42.155412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:42.155758 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:42.155959 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:42.155967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:42.157269 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:42.157274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:42.158570 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:42.158581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:42.159304 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:42.159308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:42.159426 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:42.160100 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:42.160117 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:42.163351 (+ 3234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.182.190:33023: Server oryx/127.1.182.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":259,"threads_started":1}
W20260928 02:05:42.163789 10726 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:42.154519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:56237 (local address 127.1.182.190:33023)
0928 02:05:42.154762 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:42.154771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:42.155010 (+ 239us) server_negotiation.cc:407] Connection header received
0928 02:05:42.155156 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:42.155159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:42.155278 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:42.155364 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:42.156118 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:42.157077 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:42.158696 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:42.159189 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:42.159287 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:42.163539 (+ 4252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.182.130:56237: BlockingRecv error: recv got EOF from 127.1.182.130:56237 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:05:42.552181 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 10575
I20260928 02:05:42.565573 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:42.553468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37444 (local address 127.1.182.129:40179)
0928 02:05:42.554110 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:42.554118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:42.554146 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:05:42.554207 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:42.554214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:42.554558 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:42.554740 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:42.556385 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:42.557464 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:42.558696 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:42.559354 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:42.559463 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:42.562349 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:42.562358 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:42.562372 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:42.562426 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:42.565074 (+ 2648us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.182.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 02:05:42.565361 (+ 287us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37444: Request ticket server kudu/127.1.182.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":352,"threads_started":1}
W20260928 02:05:42.565748 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 10575: Not authorized: Client connection negotiation failed: client connection to 127.1.182.129:40179: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.182.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 02:05:42.565980 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 10575
I20260928 02:05:42.582877 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 10728
I20260928 02:05:42.595865 11041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:42.584020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.1.182.130:40979)
0928 02:05:42.584643 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:42.584652 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:42.584687 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:05:42.584769 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:42.584777 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:42.585082 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:42.585300 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:42.586541 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:42.587676 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:42.588836 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:42.589423 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:42.589522 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:42.592239 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:42.592250 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:42.592267 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:42.592310 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:42.595036 (+ 2726us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.182.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 02:05:42.595553 (+ 517us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60840: Request ticket server kudu/127.1.182.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":279,"threads_started":1}
W20260928 02:05:42.595933 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 10728: Not authorized: Client connection negotiation failed: client connection to 127.1.182.130:40979: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.182.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 02:05:42.595985 11041 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60840: Request ticket server kudu/127.1.182.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 02:05:42.596094 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 10728
I20260928 02:05:42.612991 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 10862
I20260928 02:05:42.626096 11042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:42.614223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43470 (local address 127.1.182.131:44457)
0928 02:05:42.614967 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:42.614974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:42.615001 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:05:42.615122 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:42.615134 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:42.615429 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:42.615643 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:42.617027 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:42.618016 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:42.619334 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:42.619872 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:42.620049 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:42.622497 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:42.622505 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:42.622519 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:42.622566 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:42.625409 (+ 2843us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.182.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 02:05:42.625747 (+ 338us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43470: Request ticket server kudu/127.1.182.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":372,"threads_started":1}
W20260928 02:05:42.626310 11042 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43470: Request ticket server kudu/127.1.182.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 02:05:42.626324 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 10862: Not authorized: Client connection negotiation failed: client connection to 127.1.182.131:44457: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.182.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 02:05:42.626531 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 10862
I20260928 02:05:42.647336 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 10506
I20260928 02:05:42.649303 10568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:42.780973 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 10506
2026-09-28T02:05:42Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21406 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T02:05:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:05:42Z Disabled control of system clock
I20260928 02:05:42.858632 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:38285
--webserver_interface=127.1.182.190
--webserver_port=0
--builtin_ntp_servers=127.1.182.148:39443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:38285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:05:43.181315 11047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:43.181635 11047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:43.181715 11047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:43.189033 11047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:05:43.189141 11047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:43.189179 11047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:43.189229 11047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:05:43.189280 11047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:05:43.198374 11047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:38285
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:38285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:43.199659 11047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:43.201390 11047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:43.213872 11052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:43.213872 11053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:43.213905 11055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:43.214644 11047 server_base.cc:1034] running on GCE node
I20260928 02:05:43.215631 11047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:43.217089 11047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:43.218369 11047 hybrid_clock.cc:648] HybridClock initialized: now 1790561143218327 us; error 55 us; skew 500 ppm
I20260928 02:05:43.218708 11047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:05:43.226186 11047 webserver.cc:460] Webserver started at http://127.1.182.190:44859/ using document root <none> and password file <none>
I20260928 02:05:43.226913 11047 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:43.227033 11047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:43.227406 11047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:43.230212 11047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "41d599a21829451bbafac92e9464fdbf"
format_stamp: "Formatted at 2026-09-28 02:05:43 on dist-test-slave-23bp"
server_key: "53b8d14a01a2c3c6e21290d0dac6b634"
server_key_iv: "d49422073aec780493d7fd9121c64d48"
server_key_version: "encryptionkey@0"
I20260928 02:05:43.230916 11047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "41d599a21829451bbafac92e9464fdbf"
format_stamp: "Formatted at 2026-09-28 02:05:43 on dist-test-slave-23bp"
server_key: "53b8d14a01a2c3c6e21290d0dac6b634"
server_key_iv: "d49422073aec780493d7fd9121c64d48"
server_key_version: "encryptionkey@0"
I20260928 02:05:43.236179 11047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 02:05:43.239647 11061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:43.241170 11047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:05:43.241354 11047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "41d599a21829451bbafac92e9464fdbf"
format_stamp: "Formatted at 2026-09-28 02:05:43 on dist-test-slave-23bp"
server_key: "53b8d14a01a2c3c6e21290d0dac6b634"
server_key_iv: "d49422073aec780493d7fd9121c64d48"
server_key_version: "encryptionkey@0"
I20260928 02:05:43.241585 11047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:43.275589 11047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:43.276491 11047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:43.276791 11047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:43.293848 11047 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:38285
I20260928 02:05:43.293870 11113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:38285 every 8 connection(s)
I20260928 02:05:43.295288 11047 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:05:43.298955 11114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:43.299818 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 11047
I20260928 02:05:43.300015 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:05:43.300705 1754 external_mini_cluster.cc:1428] Setting key 7992fb602b88e9ecc838bafaf0ec9c1e
I20260928 02:05:43.311491 11114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf: Bootstrap starting.
I20260928 02:05:43.314136 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:43.302807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37208 (local address 127.1.182.190:38285)
0928 02:05:43.303859 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:43.303872 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:43.303920 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:05:43.305416 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:43.305437 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:43.306001 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:43.306418 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:43.306955 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:43.308294 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:43.309610 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:43.310265 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:43.310469 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:43.311745 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:43.311753 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:43.311777 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:05:43.312416 (+ 639us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:43.312500 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:05:43.312520 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:43.312660 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:43.312890 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 02:05:43.313175 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":387,"threads_started":1}
I20260928 02:05:43.314682 11114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:43.315816 11114 log.cc:824] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf: Log is configured to *not* fsync() on all Append() calls
I20260928 02:05:43.318543 11114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf: No bootstrap required, opened a new log
I20260928 02:05:43.325626 11114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d599a21829451bbafac92e9464fdbf" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38285 } }
I20260928 02:05:43.325977 11114 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:43.326073 11114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41d599a21829451bbafac92e9464fdbf, State: Initialized, Role: FOLLOWER
I20260928 02:05:43.326675 11114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d599a21829451bbafac92e9464fdbf" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38285 } }
I20260928 02:05:43.326905 11114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:05:43.327028 11114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:05:43.327227 11114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:43.329599 11114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d599a21829451bbafac92e9464fdbf" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38285 } }
I20260928 02:05:43.330072 11114 leader_election.cc:302] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41d599a21829451bbafac92e9464fdbf; no voters:
I20260928 02:05:43.330634 11114 leader_election.cc:288] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:05:43.330758 11119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:05:43.331476 11119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf [term 1 LEADER]: Becoming Leader. State: Replica: 41d599a21829451bbafac92e9464fdbf, State: Running, Role: LEADER
I20260928 02:05:43.332002 11119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d599a21829451bbafac92e9464fdbf" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38285 } }
I20260928 02:05:43.332770 11114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:05:43.334875 11120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41d599a21829451bbafac92e9464fdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d599a21829451bbafac92e9464fdbf" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38285 } } }
I20260928 02:05:43.335531 11120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:43.334900 11121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41d599a21829451bbafac92e9464fdbf. Latest consensus state: current_term: 1 leader_uuid: "41d599a21829451bbafac92e9464fdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d599a21829451bbafac92e9464fdbf" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38285 } } }
I20260928 02:05:43.335944 11121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:43.337659 11127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:05:43.342985 11127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:05:43.352090 11127 catalog_manager.cc:1347] Generated new cluster ID: b5b55c1812c64638a226eb10ba6a44a3
I20260928 02:05:43.352226 11127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:05:43.369216 11127 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:05:43.370440 11127 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:05:43.381140 11127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41d599a21829451bbafac92e9464fdbf: Generated new TSK 0
I20260928 02:05:43.381947 11127 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:05:43.386636 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:38285
--builtin_ntp_servers=127.1.182.148:39443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 02:05:43.724800 11138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:43.725064 11138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:43.725148 11138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:43.732331 11138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:43.732424 11138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:43.732524 11138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:05:43.741706 11138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:38285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:43.742901 11138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:43.744583 11138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:43.757596 11143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:43.757557 11144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:43.757638 11146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:43.758194 11138 server_base.cc:1034] running on GCE node
I20260928 02:05:43.758772 11138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:43.759778 11138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:43.761039 11138 hybrid_clock.cc:648] HybridClock initialized: now 1790561143760994 us; error 55 us; skew 500 ppm
I20260928 02:05:43.761552 11138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:05:43.763854 11138 webserver.cc:460] Webserver started at http://127.1.182.129:34943/ using document root <none> and password file <none>
I20260928 02:05:43.764623 11138 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:43.764772 11138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:43.765101 11138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:43.767997 11138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "4cfe521dc6f84b189706deb70874c321"
format_stamp: "Formatted at 2026-09-28 02:05:43 on dist-test-slave-23bp"
server_key: "bd5e118479949b9fae3882957014f64a"
server_key_iv: "cc55e1058ebc2dcda5e145f5f438ffe0"
server_key_version: "encryptionkey@0"
I20260928 02:05:43.768788 11138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "4cfe521dc6f84b189706deb70874c321"
format_stamp: "Formatted at 2026-09-28 02:05:43 on dist-test-slave-23bp"
server_key: "bd5e118479949b9fae3882957014f64a"
server_key_iv: "cc55e1058ebc2dcda5e145f5f438ffe0"
server_key_version: "encryptionkey@0"
I20260928 02:05:43.773825 11138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:05:43.777179 11152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:43.778487 11138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:05:43.778666 11138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "4cfe521dc6f84b189706deb70874c321"
format_stamp: "Formatted at 2026-09-28 02:05:43 on dist-test-slave-23bp"
server_key: "bd5e118479949b9fae3882957014f64a"
server_key_iv: "cc55e1058ebc2dcda5e145f5f438ffe0"
server_key_version: "encryptionkey@0"
I20260928 02:05:43.778900 11138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:43.792213 11138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:43.793053 11138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:43.793358 11138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:43.794236 11138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:43.795769 11138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:43.795886 11138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:43.796023 11138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:43.796113 11138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:43.829252 11138 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:35271
I20260928 02:05:43.829303 11266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:35271 every 8 connection(s)
I20260928 02:05:43.830590 11138 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:05:43.838608 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 11138
I20260928 02:05:43.838800 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:05:43.839524 1754 external_mini_cluster.cc:1428] Setting key 97743bae53beb1b58412a8bf5a3edc60
I20260928 02:05:43.842233 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:38285
--builtin_ntp_servers=127.1.182.148:39443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 02:05:43.843439 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:43.833252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:47645 (local address 127.1.182.190:38285)
0928 02:05:43.833733 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:43.833738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:43.835055 (+ 1317us) server_negotiation.cc:407] Connection header received
0928 02:05:43.836855 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:43.836861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:43.836988 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:43.837124 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:43.838181 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:43.839031 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:43.840623 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:43.841028 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:43.841158 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:43.842579 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:43.842583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:43.842586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:05:43.842602 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:43.842647 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:05:43.842659 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:43.842757 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:43.842993 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 02:05:43.843111 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":207,"threads_started":1}
I20260928 02:05:43.844491 11270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:43.834005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:38285 (local address 127.1.182.129:47645)
0928 02:05:43.834899 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:43.834943 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:43.836575 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:43.837355 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:43.837363 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:43.837449 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:43.838005 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:43.838020 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:43.839194 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:43.839206 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:43.840480 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:43.840491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:43.841271 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:43.841279 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:43.841502 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:43.842219 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:05:43.842222 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:43.842261 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:05:43.842376 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:43.842755 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:43.842767 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:43.842805 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:05:43.842985 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 02:05:43.843399 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":274,"threads_started":1}
I20260928 02:05:43.846125 11267 heartbeater.cc:347] Connected to a master server at 127.1.182.190:38285
I20260928 02:05:43.846449 11267 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:43.847476 11267 heartbeater.cc:511] Master 127.1.182.190:38285 requested a full tablet report, sending...
I20260928 02:05:43.850147 11079 ts_manager.cc:194] Registered new tserver with Master: 4cfe521dc6f84b189706deb70874c321 (127.1.182.129:35271)
I20260928 02:05:43.852073 11079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.182.129:47645
W20260928 02:05:44.169420 11271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:44.169740 11271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:44.169912 11271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:44.176967 11271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:44.177111 11271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:44.177304 11271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:05:44.186386 11271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:38285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:44.187705 11271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:44.189486 11271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:44.202430 11276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:44.202440 11277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:44.203060 11279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:44.203508 11271 server_base.cc:1034] running on GCE node
I20260928 02:05:44.204166 11271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:44.205178 11271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:44.206424 11271 hybrid_clock.cc:648] HybridClock initialized: now 1790561144206373 us; error 47 us; skew 500 ppm
I20260928 02:05:44.206770 11271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:05:44.209199 11271 webserver.cc:460] Webserver started at http://127.1.182.130:41217/ using document root <none> and password file <none>
I20260928 02:05:44.209914 11271 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:44.210059 11271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:44.210450 11271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:44.213282 11271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "d3b0f32388124370aba3743ddab7f8fc"
format_stamp: "Formatted at 2026-09-28 02:05:44 on dist-test-slave-23bp"
server_key: "3e8bb7ef4774b5d0bc89c36b281c4561"
server_key_iv: "55a31d47b058a94920d3f052dd1ed75a"
server_key_version: "encryptionkey@0"
I20260928 02:05:44.214069 11271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "d3b0f32388124370aba3743ddab7f8fc"
format_stamp: "Formatted at 2026-09-28 02:05:44 on dist-test-slave-23bp"
server_key: "3e8bb7ef4774b5d0bc89c36b281c4561"
server_key_iv: "55a31d47b058a94920d3f052dd1ed75a"
server_key_version: "encryptionkey@0"
I20260928 02:05:44.219187 11271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 02:05:44.222716 11285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:44.224258 11271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 02:05:44.224483 11271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "d3b0f32388124370aba3743ddab7f8fc"
format_stamp: "Formatted at 2026-09-28 02:05:44 on dist-test-slave-23bp"
server_key: "3e8bb7ef4774b5d0bc89c36b281c4561"
server_key_iv: "55a31d47b058a94920d3f052dd1ed75a"
server_key_version: "encryptionkey@0"
I20260928 02:05:44.224720 11271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:44.263801 11271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:44.264724 11271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:44.265049 11271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:44.266036 11271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:44.267581 11271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:44.267716 11271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:44.267868 11271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:44.267964 11271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:44.304211 11271 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:36263
I20260928 02:05:44.304308 11398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:36263 every 8 connection(s)
I20260928 02:05:44.305800 11271 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:05:44.312357 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 11271
I20260928 02:05:44.312546 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:05:44.313069 1754 external_mini_cluster.cc:1428] Setting key 14a19dc56d5e9ffa96a3e94102366f4b
I20260928 02:05:44.316025 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:38285
--builtin_ntp_servers=127.1.182.148:39443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 02:05:44.320150 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:44.308696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:45561 (local address 127.1.182.190:38285)
0928 02:05:44.308896 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:44.308902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:44.310419 (+ 1517us) server_negotiation.cc:407] Connection header received
0928 02:05:44.312937 (+ 2518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:44.312943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:44.313024 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:44.313147 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:44.314486 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:44.315365 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:44.317013 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:44.317743 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:44.317898 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:44.319241 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:44.319250 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:44.319254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:05:44.319277 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:44.319318 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:05:44.319337 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:44.319431 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:44.319653 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 02:05:44.319763 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:05:44.321007 11401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:44.309205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:38285 (local address 127.1.182.130:45561)
0928 02:05:44.310241 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:44.310293 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:44.312611 (+ 2318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:44.313418 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:44.313427 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:44.313515 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:44.314314 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:44.314338 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:44.315516 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:44.315520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:44.316861 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:44.316873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:44.317961 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:44.317966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:44.318160 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:44.318894 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:05:44.318897 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:44.318936 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:05:44.319014 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:44.319439 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:44.319447 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:44.319453 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:05:44.319605 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 02:05:44.319910 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":267,"threads_started":1}
I20260928 02:05:44.322559 11399 heartbeater.cc:347] Connected to a master server at 127.1.182.190:38285
I20260928 02:05:44.323021 11399 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:44.324093 11399 heartbeater.cc:511] Master 127.1.182.190:38285 requested a full tablet report, sending...
I20260928 02:05:44.325629 11079 ts_manager.cc:194] Registered new tserver with Master: d3b0f32388124370aba3743ddab7f8fc (127.1.182.130:36263)
I20260928 02:05:44.326579 11079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.182.130:45561
W20260928 02:05:44.644871 11402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:44.645229 11402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:44.645395 11402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:44.653028 11402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:44.653180 11402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:44.653363 11402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:05:44.662762 11402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:38285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:44.664130 11402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:44.665920 11402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:44.679337 11408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:44.679366 11410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:44.679581 11407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:44.679807 11402 server_base.cc:1034] running on GCE node
I20260928 02:05:44.680734 11402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:44.681723 11402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:44.682951 11402 hybrid_clock.cc:648] HybridClock initialized: now 1790561144682898 us; error 50 us; skew 500 ppm
I20260928 02:05:44.683305 11402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:05:44.685688 11402 webserver.cc:460] Webserver started at http://127.1.182.131:33437/ using document root <none> and password file <none>
I20260928 02:05:44.686415 11402 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:44.686548 11402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:44.686877 11402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:44.689756 11402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "f308b2dd323b481588e1d903d0a6a9b2"
format_stamp: "Formatted at 2026-09-28 02:05:44 on dist-test-slave-23bp"
server_key: "5eb0929a3b5c987a70da609f36797f39"
server_key_iv: "ea06af68ee2c9a5fafae772f8d084202"
server_key_version: "encryptionkey@0"
I20260928 02:05:44.690538 11402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "f308b2dd323b481588e1d903d0a6a9b2"
format_stamp: "Formatted at 2026-09-28 02:05:44 on dist-test-slave-23bp"
server_key: "5eb0929a3b5c987a70da609f36797f39"
server_key_iv: "ea06af68ee2c9a5fafae772f8d084202"
server_key_version: "encryptionkey@0"
I20260928 02:05:44.695634 11402 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 02:05:44.698908 11416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:44.700358 11402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:05:44.700532 11402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "f308b2dd323b481588e1d903d0a6a9b2"
format_stamp: "Formatted at 2026-09-28 02:05:44 on dist-test-slave-23bp"
server_key: "5eb0929a3b5c987a70da609f36797f39"
server_key_iv: "ea06af68ee2c9a5fafae772f8d084202"
server_key_version: "encryptionkey@0"
I20260928 02:05:44.700738 11402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:44.728876 11402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:44.729755 11402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:44.730064 11402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:44.731002 11402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:44.732451 11402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:44.732570 11402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:44.732708 11402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:44.732801 11402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:44.770648 11402 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:43257
I20260928 02:05:44.770720 11530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:43257 every 8 connection(s)
I20260928 02:05:44.772006 11402 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:05:44.781296 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 11402
I20260928 02:05:44.781478 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:05:44.782008 1754 external_mini_cluster.cc:1428] Setting key 749ab8b01176b2505af04ab51c535513
I20260928 02:05:44.785005 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:44.774681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:50299 (local address 127.1.182.190:38285)
0928 02:05:44.774870 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:44.774875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:44.776269 (+ 1394us) server_negotiation.cc:407] Connection header received
0928 02:05:44.777974 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:44.777978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:44.778045 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:44.778138 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:44.779225 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:44.779956 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:44.781808 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:44.782282 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:44.782391 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:44.784149 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:44.784154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:44.784157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:05:44.784175 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:44.784207 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:05:44.784219 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:44.784331 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:44.784554 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 02:05:44.784653 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:05:44.785976 11533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:44.775243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:38285 (local address 127.1.182.131:50299)
0928 02:05:44.776123 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:44.776158 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:44.777701 (+ 1543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:44.778335 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:44.778347 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:44.778442 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:44.779018 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:44.779033 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:44.780082 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:44.780085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:44.781643 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:44.781657 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:44.782521 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:44.782531 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:44.782875 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:44.783756 (+ 881us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:05:44.783760 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:44.783817 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:05:44.783942 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:44.784346 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:44.784355 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:44.784361 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:05:44.784505 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 02:05:44.784798 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":245,"threads_started":1}
I20260928 02:05:44.787257 11531 heartbeater.cc:347] Connected to a master server at 127.1.182.190:38285
I20260928 02:05:44.787585 11531 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:44.788308 11531 heartbeater.cc:511] Master 127.1.182.190:38285 requested a full tablet report, sending...
I20260928 02:05:44.789629 11079 ts_manager.cc:194] Registered new tserver with Master: f308b2dd323b481588e1d903d0a6a9b2 (127.1.182.131:43257)
I20260928 02:05:44.790443 11079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.182.131:50299
I20260928 02:05:44.797859 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:05:44.804575 11269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:44.802659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37220 (local address 127.1.182.190:38285)
0928 02:05:44.802863 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:44.802868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:44.803504 (+ 636us) server_negotiation.cc:407] Connection header received
0928 02:05:44.803698 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:44.803701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:44.803784 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:44.803911 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:44.804354 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37220: BlockingRecv error: recv got EOF from 127.0.0.1:37220 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:05:44.805583 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 11138
I20260928 02:05:44.813846 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:44.806652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34930 (local address 127.1.182.129:35271)
0928 02:05:44.807221 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:44.807231 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:44.807271 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:05:44.807332 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:44.807338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:44.807640 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:44.807790 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:44.808116 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:44.809341 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:44.810680 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:44.811365 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:44.811471 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:44.812465 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:44.812473 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:44.812486 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:05:44.813073 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:44.813118 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:05:44.813142 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:44.813238 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:44.813430 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 02:05:44.813551 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":286,"threads_started":1}
I20260928 02:05:44.815055 11262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:44.898321 11267 heartbeater.cc:503] Master 127.1.182.190:38285 was elected leader, sending a full tablet report...
I20260928 02:05:44.899333 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 11138
W20260928 02:05:44.915795 11062 connection.cc:535] server connection from 127.1.182.129:47645 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:05:44.916139 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 11271
I20260928 02:05:44.924465 11542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:44.917106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45764 (local address 127.1.182.130:36263)
0928 02:05:44.917585 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:44.917593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:44.917619 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:05:44.917681 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:44.917687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:44.917908 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:44.918040 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:44.918443 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:44.919532 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:44.920825 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:44.921433 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:44.921529 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:44.922633 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:44.922641 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:44.922655 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:05:44.923540 (+ 885us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:44.923588 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:05:44.923623 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:44.923716 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:44.923895 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 02:05:44.924042 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":254,"threads_started":1}
I20260928 02:05:44.925777 11394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:45.007992 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 11271
I20260928 02:05:45.024319 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 11402
I20260928 02:05:45.032797 11544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:45.025338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37562 (local address 127.1.182.131:43257)
0928 02:05:45.025928 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:45.025934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:45.025964 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:05:45.026021 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:45.026028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:45.026247 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:45.026427 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:45.026891 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:45.028024 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:45.029322 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:45.029881 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:45.030020 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:45.031249 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:45.031259 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:45.031320 (+ 61us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:05:45.031920 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:45.031966 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:05:45.031985 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:45.032111 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:45.032290 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 02:05:45.032424 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":281,"threads_started":1}
I20260928 02:05:45.034338 11526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:45.115736 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 11402
I20260928 02:05:45.132035 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 11047
I20260928 02:05:45.133381 11109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:45.233532 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 11047
2026-09-28T02:05:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2449 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T02:05:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:05:45Z Disabled control of system clock
I20260928 02:05:45.308864 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:37221
--webserver_interface=127.1.182.190
--webserver_port=0
--builtin_ntp_servers=127.1.182.148:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:37221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:05:45.630673 11550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:45.631083 11550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:45.631304 11550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:45.639760 11550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:05:45.639892 11550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:45.639956 11550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:45.640041 11550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:05:45.640117 11550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:05:45.649663 11550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:37221
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:37221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:45.651039 11550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:45.652865 11550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:45.665061 11555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:45.665089 11558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:45.665203 11556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:45.665887 11550 server_base.cc:1034] running on GCE node
I20260928 02:05:45.666677 11550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:45.668176 11550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:45.669389 11550 hybrid_clock.cc:648] HybridClock initialized: now 1790561145669350 us; error 35 us; skew 500 ppm
I20260928 02:05:45.669736 11550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:05:45.672139 11550 webserver.cc:460] Webserver started at http://127.1.182.190:40243/ using document root <none> and password file <none>
I20260928 02:05:45.672886 11550 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:45.672999 11550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:45.673343 11550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:45.676313 11550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "6296ac9ede2b4b268ef6252ef90c8c89"
format_stamp: "Formatted at 2026-09-28 02:05:45 on dist-test-slave-23bp"
server_key: "ec52e997a89b9146e5c831c348019ff3"
server_key_iv: "6785246eba7e59eeca1b107cd5cc2fbc"
server_key_version: "encryptionkey@0"
I20260928 02:05:45.677098 11550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "6296ac9ede2b4b268ef6252ef90c8c89"
format_stamp: "Formatted at 2026-09-28 02:05:45 on dist-test-slave-23bp"
server_key: "ec52e997a89b9146e5c831c348019ff3"
server_key_iv: "6785246eba7e59eeca1b107cd5cc2fbc"
server_key_version: "encryptionkey@0"
I20260928 02:05:45.682235 11550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:05:45.685891 11564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:45.687407 11550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:05:45.687575 11550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "6296ac9ede2b4b268ef6252ef90c8c89"
format_stamp: "Formatted at 2026-09-28 02:05:45 on dist-test-slave-23bp"
server_key: "ec52e997a89b9146e5c831c348019ff3"
server_key_iv: "6785246eba7e59eeca1b107cd5cc2fbc"
server_key_version: "encryptionkey@0"
I20260928 02:05:45.687772 11550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:45.693722 11550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:45.694527 11550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:45.694830 11550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:45.713495 11550 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:37221
I20260928 02:05:45.713543 11615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:37221 every 8 connection(s)
I20260928 02:05:45.714830 11550 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:05:45.718600 11616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:45.720681 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 11550
I20260928 02:05:45.720920 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:05:45.721726 1754 external_mini_cluster.cc:1428] Setting key c678c3bd82b1bb6ccfe21be9622bb5d9
I20260928 02:05:45.729770 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:45.723692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51730 (local address 127.1.182.190:37221)
0928 02:05:45.724721 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:45.724732 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:45.724782 (+ 50us) server_negotiation.cc:407] Connection header received
0928 02:05:45.726273 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:45.726295 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:45.726725 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:45.727046 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:45.728167 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:45.728173 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:45.728186 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:05:45.728198 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:45.728259 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:05:45.728275 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:45.728439 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:45.728565 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 02:05:45.728877 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":393,"threads_started":1}
I20260928 02:05:45.733197 11616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89: Bootstrap starting.
I20260928 02:05:45.736099 11616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:45.737072 11616 log.cc:824] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89: Log is configured to *not* fsync() on all Append() calls
I20260928 02:05:45.740552 11616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89: No bootstrap required, opened a new log
I20260928 02:05:45.749910 11616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6296ac9ede2b4b268ef6252ef90c8c89" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37221 } }
I20260928 02:05:45.750283 11616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:45.750413 11616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6296ac9ede2b4b268ef6252ef90c8c89, State: Initialized, Role: FOLLOWER
I20260928 02:05:45.751020 11616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6296ac9ede2b4b268ef6252ef90c8c89" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37221 } }
I20260928 02:05:45.751317 11616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:05:45.751469 11616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:05:45.751637 11616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:45.753661 11616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6296ac9ede2b4b268ef6252ef90c8c89" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37221 } }
I20260928 02:05:45.754133 11616 leader_election.cc:302] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6296ac9ede2b4b268ef6252ef90c8c89; no voters:
I20260928 02:05:45.754763 11616 leader_election.cc:288] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:05:45.754925 11621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:05:45.755690 11621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89 [term 1 LEADER]: Becoming Leader. State: Replica: 6296ac9ede2b4b268ef6252ef90c8c89, State: Running, Role: LEADER
I20260928 02:05:45.756335 11621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6296ac9ede2b4b268ef6252ef90c8c89" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37221 } }
I20260928 02:05:45.756799 11616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:05:45.759181 11622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6296ac9ede2b4b268ef6252ef90c8c89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6296ac9ede2b4b268ef6252ef90c8c89" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37221 } } }
I20260928 02:05:45.759624 11622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:45.760151 11623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6296ac9ede2b4b268ef6252ef90c8c89. Latest consensus state: current_term: 1 leader_uuid: "6296ac9ede2b4b268ef6252ef90c8c89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6296ac9ede2b4b268ef6252ef90c8c89" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37221 } } }
I20260928 02:05:45.760519 11623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:45.761153 11629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:05:45.766172 11629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:05:45.774539 11629 catalog_manager.cc:1347] Generated new cluster ID: 62c0505f5a404d42b8e0d0ba95993395
I20260928 02:05:45.774641 11629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:05:45.791433 11629 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:05:45.791860 11629 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:05:45.800467 11629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6296ac9ede2b4b268ef6252ef90c8c89: Generated new TSK 0
I20260928 02:05:45.801430 11629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:05:45.811772 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:37221
--builtin_ntp_servers=127.1.182.148:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:05:46.142416 11640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:46.142680 11640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:46.142845 11640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:46.150142 11640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:46.150238 11640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:46.150434 11640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:05:46.159739 11640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:37221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:46.161024 11640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:46.163023 11640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:46.176743 11645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:46.176688 11648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:46.176753 11646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:46.177627 11640 server_base.cc:1034] running on GCE node
I20260928 02:05:46.178194 11640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:46.179257 11640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:46.180541 11640 hybrid_clock.cc:648] HybridClock initialized: now 1790561146180476 us; error 61 us; skew 500 ppm
I20260928 02:05:46.180883 11640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:05:46.183368 11640 webserver.cc:460] Webserver started at http://127.1.182.129:37975/ using document root <none> and password file <none>
I20260928 02:05:46.184064 11640 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:46.184192 11640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:46.184525 11640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:46.187506 11640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "8b08e39fb59e40dc93dc78c869eadb87"
format_stamp: "Formatted at 2026-09-28 02:05:46 on dist-test-slave-23bp"
server_key: "3ac7f04b38ebec686829a797bb12e581"
server_key_iv: "021d94580f31973720e4682628a6a8db"
server_key_version: "encryptionkey@0"
I20260928 02:05:46.188295 11640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "8b08e39fb59e40dc93dc78c869eadb87"
format_stamp: "Formatted at 2026-09-28 02:05:46 on dist-test-slave-23bp"
server_key: "3ac7f04b38ebec686829a797bb12e581"
server_key_iv: "021d94580f31973720e4682628a6a8db"
server_key_version: "encryptionkey@0"
I20260928 02:05:46.193543 11640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 02:05:46.197009 11654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:46.198345 11640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:05:46.198519 11640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "8b08e39fb59e40dc93dc78c869eadb87"
format_stamp: "Formatted at 2026-09-28 02:05:46 on dist-test-slave-23bp"
server_key: "3ac7f04b38ebec686829a797bb12e581"
server_key_iv: "021d94580f31973720e4682628a6a8db"
server_key_version: "encryptionkey@0"
I20260928 02:05:46.198731 11640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:46.204528 11640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:46.205335 11640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:46.205629 11640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:46.206552 11640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:46.207998 11640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:46.208114 11640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:46.208245 11640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:46.208333 11640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:46.240737 11640 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:37097
I20260928 02:05:46.240782 11767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:37097 every 8 connection(s)
I20260928 02:05:46.242162 11640 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:05:46.246331 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 11640
I20260928 02:05:46.246539 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:05:46.247221 1754 external_mini_cluster.cc:1428] Setting key 10edda6112c1c64242038dbd9138cfab
I20260928 02:05:46.250008 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:37221
--builtin_ntp_servers=127.1.182.148:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 02:05:46.251056 11770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:46.244903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:47023 (local address 127.1.182.190:37221)
0928 02:05:46.245338 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:46.245342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:46.246500 (+ 1158us) server_negotiation.cc:407] Connection header received
0928 02:05:46.248401 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:46.248407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:46.248472 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:46.248599 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:46.250044 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:46.250049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:46.250054 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:05:46.250079 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:46.250118 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:05:46.250154 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:46.250337 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:46.250609 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 02:05:46.250676 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":231,"threads_started":1}
I20260928 02:05:46.251869 11771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:46.245387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:37221 (local address 127.1.182.129:47023)
0928 02:05:46.246298 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:46.246347 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:46.248139 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:46.248876 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:46.248887 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:46.249010 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:46.249724 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:05:46.249727 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:46.249771 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:05:46.249855 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:46.250384 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:46.250392 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:46.250398 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:05:46.250622 (+ 224us) client_negotiation.cc:240] Negotiation successful
0928 02:05:46.250823 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":278,"threads_started":1}
I20260928 02:05:46.253302 11768 heartbeater.cc:347] Connected to a master server at 127.1.182.190:37221
I20260928 02:05:46.253600 11768 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:46.254142 11768 heartbeater.cc:511] Master 127.1.182.190:37221 requested a full tablet report, sending...
I20260928 02:05:46.256127 11581 ts_manager.cc:194] Registered new tserver with Master: 8b08e39fb59e40dc93dc78c869eadb87 (127.1.182.129:37097)
W20260928 02:05:46.574704 11772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:46.574973 11772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:46.575155 11772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:46.582525 11772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:46.582623 11772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:46.582799 11772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:05:46.592048 11772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:37221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:46.593452 11772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:46.595338 11772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:46.611719 11777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:46.612138 11778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:46.612593 11780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:46.613073 11772 server_base.cc:1034] running on GCE node
I20260928 02:05:46.613803 11772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:46.614794 11772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:46.616031 11772 hybrid_clock.cc:648] HybridClock initialized: now 1790561146615966 us; error 61 us; skew 500 ppm
I20260928 02:05:46.616391 11772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:05:46.618577 11772 webserver.cc:460] Webserver started at http://127.1.182.130:38719/ using document root <none> and password file <none>
I20260928 02:05:46.619338 11772 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:46.619467 11772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:46.619798 11772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:46.622625 11772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "39899e3e2b924ff6b8739c75bfd9bf3c"
format_stamp: "Formatted at 2026-09-28 02:05:46 on dist-test-slave-23bp"
server_key: "a071c375e608bbc6448cc64a93404877"
server_key_iv: "2c72d9d101fa220483bd3dc2e3fbbaf0"
server_key_version: "encryptionkey@0"
I20260928 02:05:46.623466 11772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "39899e3e2b924ff6b8739c75bfd9bf3c"
format_stamp: "Formatted at 2026-09-28 02:05:46 on dist-test-slave-23bp"
server_key: "a071c375e608bbc6448cc64a93404877"
server_key_iv: "2c72d9d101fa220483bd3dc2e3fbbaf0"
server_key_version: "encryptionkey@0"
I20260928 02:05:46.629717 11772 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 02:05:46.633226 11786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:46.634536 11772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:05:46.634706 11772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "39899e3e2b924ff6b8739c75bfd9bf3c"
format_stamp: "Formatted at 2026-09-28 02:05:46 on dist-test-slave-23bp"
server_key: "a071c375e608bbc6448cc64a93404877"
server_key_iv: "2c72d9d101fa220483bd3dc2e3fbbaf0"
server_key_version: "encryptionkey@0"
I20260928 02:05:46.634930 11772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:46.640938 11772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:46.641916 11772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:46.642174 11772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:46.643297 11772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:46.644913 11772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:46.645026 11772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:46.645133 11772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:46.645226 11772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:46.677888 11772 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:33761
I20260928 02:05:46.677932 11899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:33761 every 8 connection(s)
I20260928 02:05:46.679443 11772 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:05:46.683673 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 11772
I20260928 02:05:46.683866 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:05:46.684427 1754 external_mini_cluster.cc:1428] Setting key 8a5be95fcc2291ec6ea6ec60b96a625d
I20260928 02:05:46.687536 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:37221
--builtin_ntp_servers=127.1.182.148:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 02:05:46.688704 11770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:46.682380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:47671 (local address 127.1.182.190:37221)
0928 02:05:46.682604 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:46.682610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:46.684377 (+ 1767us) server_negotiation.cc:407] Connection header received
0928 02:05:46.686400 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:46.686405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:46.686479 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:46.686559 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:46.688066 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:46.688070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:46.688073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:05:46.688087 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:46.688116 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:05:46.688128 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:46.688212 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:46.688455 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 02:05:46.688501 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:05:46.690004 11902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:46.682966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:37221 (local address 127.1.182.130:47671)
0928 02:05:46.684207 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:46.684255 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:46.686156 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:46.686844 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:46.686857 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:46.686978 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:46.687718 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:05:46.687723 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:46.687776 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:05:46.687895 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:46.688223 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:46.688236 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:46.688246 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:05:46.688523 (+ 277us) client_negotiation.cc:240] Negotiation successful
0928 02:05:46.688778 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":350,"threads_started":1}
I20260928 02:05:46.691474 11900 heartbeater.cc:347] Connected to a master server at 127.1.182.190:37221
I20260928 02:05:46.691804 11900 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:46.692497 11900 heartbeater.cc:511] Master 127.1.182.190:37221 requested a full tablet report, sending...
I20260928 02:05:46.693969 11581 ts_manager.cc:194] Registered new tserver with Master: 39899e3e2b924ff6b8739c75bfd9bf3c (127.1.182.130:33761)
W20260928 02:05:47.019296 11903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:47.019610 11903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:47.019785 11903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:47.026993 11903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:47.027160 11903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:47.027348 11903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:05:47.036624 11903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:37221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:47.037942 11903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:47.040038 11903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:47.053150 11908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:47.053279 11911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:47.053303 11909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:47.054564 11903 server_base.cc:1034] running on GCE node
I20260928 02:05:47.055222 11903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:47.056138 11903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:47.057375 11903 hybrid_clock.cc:648] HybridClock initialized: now 1790561147057301 us; error 37 us; skew 500 ppm
I20260928 02:05:47.057724 11903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:05:47.060284 11903 webserver.cc:460] Webserver started at http://127.1.182.131:40177/ using document root <none> and password file <none>
I20260928 02:05:47.060926 11903 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:47.061009 11903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:47.061282 11903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:47.064314 11903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "7e5f3f0857414d41aa45f1e387b55a7f"
format_stamp: "Formatted at 2026-09-28 02:05:47 on dist-test-slave-23bp"
server_key: "315b8d271b605f5ee86a93ad68a54bca"
server_key_iv: "500e8787fe102048b4f796919b8b0c35"
server_key_version: "encryptionkey@0"
I20260928 02:05:47.065096 11903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "7e5f3f0857414d41aa45f1e387b55a7f"
format_stamp: "Formatted at 2026-09-28 02:05:47 on dist-test-slave-23bp"
server_key: "315b8d271b605f5ee86a93ad68a54bca"
server_key_iv: "500e8787fe102048b4f796919b8b0c35"
server_key_version: "encryptionkey@0"
I20260928 02:05:47.070680 11903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:05:47.074708 11917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:47.076169 11903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:05:47.076354 11903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "7e5f3f0857414d41aa45f1e387b55a7f"
format_stamp: "Formatted at 2026-09-28 02:05:47 on dist-test-slave-23bp"
server_key: "315b8d271b605f5ee86a93ad68a54bca"
server_key_iv: "500e8787fe102048b4f796919b8b0c35"
server_key_version: "encryptionkey@0"
I20260928 02:05:47.076575 11903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:47.082231 11903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:47.083014 11903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:47.083321 11903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:47.084226 11903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:47.085642 11903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:47.085783 11903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:47.085918 11903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:47.086006 11903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:47.119696 11903 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:38249
I20260928 02:05:47.119788 12030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:38249 every 8 connection(s)
I20260928 02:05:47.121033 11903 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:05:47.129384 11770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:47.123786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:35007 (local address 127.1.182.190:37221)
0928 02:05:47.123975 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:47.123980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:47.125368 (+ 1388us) server_negotiation.cc:407] Connection header received
0928 02:05:47.127016 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:47.127021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:47.127156 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:47.127329 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:47.128641 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:47.128646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:47.128649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:05:47.128666 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:47.128698 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:05:47.128712 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:47.128842 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:47.129059 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 02:05:47.129109 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:05:47.130383 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:47.124332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:37221 (local address 127.1.182.131:35007)
0928 02:05:47.125225 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:47.125262 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:47.126735 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:47.127553 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:47.127562 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:47.127654 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:47.128319 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:05:47.128321 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:47.128374 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:05:47.128461 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:47.128838 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:47.128847 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:47.128854 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:05:47.129111 (+ 257us) client_negotiation.cc:240] Negotiation successful
0928 02:05:47.129341 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":260,"threads_started":1}
I20260928 02:05:47.131188 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 11903
I20260928 02:05:47.131397 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:05:47.131949 1754 external_mini_cluster.cc:1428] Setting key 1b71a70d314a7574c240b987428f61e0
I20260928 02:05:47.132073 12031 heartbeater.cc:347] Connected to a master server at 127.1.182.190:37221
I20260928 02:05:47.132443 12031 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:47.133056 12031 heartbeater.cc:511] Master 127.1.182.190:37221 requested a full tablet report, sending...
I20260928 02:05:47.134531 11580 ts_manager.cc:194] Registered new tserver with Master: 7e5f3f0857414d41aa45f1e387b55a7f (127.1.182.131:38249)
I20260928 02:05:47.147642 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:05:47.153728 11770 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:05:47.152305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51732 (local address 127.1.182.190:37221)
0928 02:05:47.152536 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:47.152544 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:47.152858 (+ 314us) server_negotiation.cc:407] Connection header received
0928 02:05:47.152985 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:47.152989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:47.153036 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:47.153118 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:47.153491 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51732: BlockingRecv error: recv got EOF from 127.0.0.1:51732 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:05:47.154496 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 11640
I20260928 02:05:47.158185 12040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:47.155538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47356 (local address 127.1.182.129:37097)
0928 02:05:47.156040 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:47.156048 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:47.156072 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:05:47.156125 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:47.156131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:47.156335 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:47.156459 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:47.157415 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:47.157421 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:47.157433 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:05:47.157444 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:47.157485 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:05:47.157499 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:47.157603 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:47.157762 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 02:05:47.157877 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":234,"threads_started":1}
I20260928 02:05:47.159471 11763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:47.241775 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 11640
I20260928 02:05:47.257813 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 11772
I20260928 02:05:47.261771 12042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:47.258892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37438 (local address 127.1.182.130:33761)
0928 02:05:47.259465 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:47.259473 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:47.259499 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:05:47.259559 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:47.259566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:47.259785 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:47.259958 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:47.260969 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:47.260982 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:47.260994 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:05:47.261013 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:47.261060 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:05:47.261074 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:47.261227 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:47.261302 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 02:05:47.261413 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":266,"threads_started":1}
I20260928 02:05:47.263375 11895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:47.350109 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 11772
I20260928 02:05:47.366871 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 11903
I20260928 02:05:47.371060 12044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:47.368006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37948 (local address 127.1.182.131:38249)
0928 02:05:47.368670 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:47.368677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:47.368707 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:05:47.368771 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:47.368779 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:47.369005 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:47.369179 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:47.370334 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:47.370341 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:47.370352 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:05:47.370368 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:47.370403 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:05:47.370417 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:47.370607 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:47.370679 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 02:05:47.370777 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":363,"threads_started":1}
I20260928 02:05:47.372336 12026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:47.453385 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 11903
I20260928 02:05:47.469470 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 11550
I20260928 02:05:47.470625 11611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:47.571822 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 11550
2026-09-28T02:05:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2334 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:05:47 dist-test-slave-23bp krb5kdc[12049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:05:47 dist-test-slave-23bp krb5kdc[12049](info): set up 2 sockets
Sep 28 02:05:47 dist-test-slave-23bp krb5kdc[12049](info): commencing operation
krb5kdc: starting...
W20260928 02:05:48.762967 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:05:48 dist-test-slave-23bp krb5kdc[12049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:05:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:05:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:05:49.000383 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:36011
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:36011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:49.323342 12062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:49.323717 12062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:49.323904 12062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:49.332020 12062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:05:49.332157 12062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:49.332267 12062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:49.332372 12062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:05:49.332448 12062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:05:49.342221 12062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:36011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:36011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:49.343621 12062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:49.344944 12062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:49.356644 12067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:49.356737 12068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:49.356777 12070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:49.357383 12062 server_base.cc:1034] running on GCE node
I20260928 02:05:49.358143 12062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:49.359616 12062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:49.360859 12062 hybrid_clock.cc:648] HybridClock initialized: now 1790561149360810 us; error 42 us; skew 500 ppm
I20260928 02:05:49.361253 12062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:49 dist-test-slave-23bp krb5kdc[12049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561149, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:49.368191 12062 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:05:49.369724 12062 webserver.cc:460] Webserver started at http://127.1.182.190:40243/ using document root <none> and password file <none>
I20260928 02:05:49.370616 12062 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:49.370793 12062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:49.371161 12062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:49.374501 12062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "3c3bca5e50cb4aeea01c5d650fd7f509"
format_stamp: "Formatted at 2026-09-28 02:05:49 on dist-test-slave-23bp"
server_key: "50552968da99d48bd128570f51f54192"
server_key_iv: "6e203b36bd999fe555e4eb5868f9b471"
server_key_version: "encryptionkey@0"
I20260928 02:05:49.375423 12062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "3c3bca5e50cb4aeea01c5d650fd7f509"
format_stamp: "Formatted at 2026-09-28 02:05:49 on dist-test-slave-23bp"
server_key: "50552968da99d48bd128570f51f54192"
server_key_iv: "6e203b36bd999fe555e4eb5868f9b471"
server_key_version: "encryptionkey@0"
I20260928 02:05:49.380553 12062 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 02:05:49.383957 12077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:49.385439 12062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:05:49.385632 12062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "3c3bca5e50cb4aeea01c5d650fd7f509"
format_stamp: "Formatted at 2026-09-28 02:05:49 on dist-test-slave-23bp"
server_key: "50552968da99d48bd128570f51f54192"
server_key_iv: "6e203b36bd999fe555e4eb5868f9b471"
server_key_version: "encryptionkey@0"
I20260928 02:05:49.385830 12062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:49.397877 12062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:49.402034 12062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:49.402308 12062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:49.421063 12062 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:36011
I20260928 02:05:49.421082 12128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:36011 every 8 connection(s)
I20260928 02:05:49.422487 12062 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:05:49.426223 12129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:49.432622 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 12062
I20260928 02:05:49.432857 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:05:49.433458 1754 external_mini_cluster.cc:1428] Setting key 7a7f0342f0b3fea1fb027d257bdf6bb8
I20260928 02:05:49.436947 12129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509: Bootstrap starting.
I20260928 02:05:49.440784 12129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:49.441828 12129 log.cc:824] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509: Log is configured to *not* fsync() on all Append() calls
I20260928 02:05:49.444571 12129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509: No bootstrap required, opened a new log
Sep 28 02:05:49 dist-test-slave-23bp krb5kdc[12049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:05:49.454551 12129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3bca5e50cb4aeea01c5d650fd7f509" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36011 } }
I20260928 02:05:49.455004 12129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:49.455176 12129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c3bca5e50cb4aeea01c5d650fd7f509, State: Initialized, Role: FOLLOWER
I20260928 02:05:49.456097 12129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3bca5e50cb4aeea01c5d650fd7f509" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36011 } }
I20260928 02:05:49.456370 12129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:05:49.456516 12129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:05:49.456713 12129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:49.457635 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:49.435492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54244 (local address 127.1.182.190:36011)
0928 02:05:49.436272 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:49.436299 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:49.436350 (+ 51us) server_negotiation.cc:407] Connection header received
0928 02:05:49.438706 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:49.438754 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:49.439288 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:49.439734 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:49.441078 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:49.442449 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:49.443799 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:49.444457 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:49.444646 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:49.448673 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:49.448683 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:49.448701 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:49.448757 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:49.452899 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:49.453575 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:49.453583 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:49.453592 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:49.453802 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:49.454143 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:49.454147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:49.454150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:49.454866 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:49.455224 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:49.455702 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 02:05:49.456166 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":329,"threads_started":1}
I20260928 02:05:49.459431 12129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3bca5e50cb4aeea01c5d650fd7f509" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36011 } }
I20260928 02:05:49.459934 12129 leader_election.cc:302] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c3bca5e50cb4aeea01c5d650fd7f509; no voters:
I20260928 02:05:49.460600 12129 leader_election.cc:288] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:05:49.460786 12134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:05:49.461611 12134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509 [term 1 LEADER]: Becoming Leader. State: Replica: 3c3bca5e50cb4aeea01c5d650fd7f509, State: Running, Role: LEADER
I20260928 02:05:49.462235 12134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3bca5e50cb4aeea01c5d650fd7f509" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36011 } }
I20260928 02:05:49.462908 12129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:05:49.465303 12135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c3bca5e50cb4aeea01c5d650fd7f509" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3bca5e50cb4aeea01c5d650fd7f509" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36011 } } }
I20260928 02:05:49.465729 12135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:49.465426 12136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c3bca5e50cb4aeea01c5d650fd7f509. Latest consensus state: current_term: 1 leader_uuid: "3c3bca5e50cb4aeea01c5d650fd7f509" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3bca5e50cb4aeea01c5d650fd7f509" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36011 } } }
I20260928 02:05:49.465847 12136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509 [sys.catalog]: This master's current role is: LEADER
I20260928 02:05:49.466763 12143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:05:49.474676 12143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:05:49.483996 12143 catalog_manager.cc:1347] Generated new cluster ID: 6be1f1c26f3042f38ef941531a3d887f
I20260928 02:05:49.484119 12143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:05:49.519465 12143 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:05:49.521420 12143 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:05:49.529810 12143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509: Generated new TSK 0
I20260928 02:05:49.530812 12143 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.182.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
I20260928 02:05:49.591162 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:36011
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:49.917065 12155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:49.917349 12155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:49.917452 12155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:49.924719 12155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:49.924832 12155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:49.924942 12155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:05:49.934340 12155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:36011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:49.935617 12155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:49.936865 12155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:49.950042 12162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:49.950186 12161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:49.950148 12164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:49.950721 12155 server_base.cc:1034] running on GCE node
I20260928 02:05:49.951411 12155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:49.952324 12155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:49.953564 12155 hybrid_clock.cc:648] HybridClock initialized: now 1790561149953511 us; error 46 us; skew 500 ppm
I20260928 02:05:49.953895 12155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:49 dist-test-slave-23bp krb5kdc[12049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561149, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:49.957340 12155 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:05:49.958839 12155 webserver.cc:460] Webserver started at http://127.1.182.129:36105/ using document root <none> and password file <none>
I20260928 02:05:49.959575 12155 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:49.959708 12155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:49.960021 12155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:49.963016 12155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "cc241e2c7f614c99a22370214eda562d"
format_stamp: "Formatted at 2026-09-28 02:05:49 on dist-test-slave-23bp"
server_key: "8918c46940be93165a23684960b38ba8"
server_key_iv: "8311c584d9e11ca0d61d9834753a3e68"
server_key_version: "encryptionkey@0"
I20260928 02:05:49.963918 12155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "cc241e2c7f614c99a22370214eda562d"
format_stamp: "Formatted at 2026-09-28 02:05:49 on dist-test-slave-23bp"
server_key: "8918c46940be93165a23684960b38ba8"
server_key_iv: "8311c584d9e11ca0d61d9834753a3e68"
server_key_version: "encryptionkey@0"
I20260928 02:05:49.969532 12155 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 02:05:49.973062 12171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:49.974470 12155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:05:49.974651 12155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "cc241e2c7f614c99a22370214eda562d"
format_stamp: "Formatted at 2026-09-28 02:05:49 on dist-test-slave-23bp"
server_key: "8918c46940be93165a23684960b38ba8"
server_key_iv: "8311c584d9e11ca0d61d9834753a3e68"
server_key_version: "encryptionkey@0"
I20260928 02:05:49.974871 12155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:50.001732 12155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:50.005792 12155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:50.006098 12155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:50.007081 12155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:50.008452 12155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:50.008555 12155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:50.008708 12155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:50.008781 12155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:50.042075 12155 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:35473
I20260928 02:05:50.042140 12284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:35473 every 8 connection(s)
I20260928 02:05:50.043460 12155 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:05:50.044008 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 12155
I20260928 02:05:50.044261 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:05:50.044991 1754 external_mini_cluster.cc:1428] Setting key a332ee436a94b93c700942634a99a182
Sep 28 02:05:50 dist-test-slave-23bp krb5kdc[12049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561149, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:05:50.064980 12287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:50.046386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:35393 (local address 127.1.182.190:36011)
0928 02:05:50.046827 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:50.046834 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:50.047987 (+ 1153us) server_negotiation.cc:407] Connection header received
0928 02:05:50.049767 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:50.049772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:50.049861 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:50.049987 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:50.051535 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:50.052240 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:50.053924 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:50.054322 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:50.054458 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:50.058277 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:50.058282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:50.058285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:50.058315 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:50.061028 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:50.061983 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:50.061987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:50.061989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:50.062107 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:50.062614 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:50.062617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:50.062619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:50.062967 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:50.063127 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:50.064141 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0928 02:05:50.064478 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":209,"threads_started":1}
I20260928 02:05:50.065558 12288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:50.047007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:36011 (local address 127.1.182.129:35393)
0928 02:05:50.047848 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:50.047885 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:50.049507 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:50.050225 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:50.050233 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:50.050716 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:50.051376 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:50.051395 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:50.052422 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:50.052426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:50.053718 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:50.053734 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:50.054545 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:50.054553 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:50.054787 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:50.055530 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:50.055552 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:50.058041 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:50.061168 (+ 3127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:50.061179 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:50.061197 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:50.061846 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:50.062217 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:50.062225 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:50.062231 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:50.062481 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:50.063114 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:50.063125 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:50.063607 (+ 482us) client_negotiation.cc:769] Sending connection context
0928 02:05:50.064009 (+ 402us) client_negotiation.cc:240] Negotiation successful
0928 02:05:50.064435 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":316,"threads_started":1}
I20260928 02:05:50.067191 12285 heartbeater.cc:347] Connected to a master server at 127.1.182.190:36011
I20260928 02:05:50.067567 12285 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.182.130@KRBTEST.COM; defaulting to no policy
I20260928 02:05:50.068475 12285 heartbeater.cc:511] Master 127.1.182.190:36011 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.130@KRBTEST.COM" created.
I20260928 02:05:50.070604 12094 ts_manager.cc:194] Registered new tserver with Master: cc241e2c7f614c99a22370214eda562d (127.1.182.129:35473)
I20260928 02:05:50.072100 12094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:35393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
I20260928 02:05:50.098498 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:36011
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:50.421729 12291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:50.422103 12291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:50.422260 12291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:50.433954 12291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:50.434087 12291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:50.434257 12291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:05:50.447464 12291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:36011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:50.448688 12291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:50.449981 12291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:50.463454 12299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:50.463454 12297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:50.463490 12296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:50.464393 12291 server_base.cc:1034] running on GCE node
I20260928 02:05:50.465010 12291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:50.465999 12291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:50.467181 12291 hybrid_clock.cc:648] HybridClock initialized: now 1790561150467133 us; error 37 us; skew 500 ppm
I20260928 02:05:50.467535 12291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:50 dist-test-slave-23bp krb5kdc[12049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561150, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:50.470849 12291 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:05:50.472843 12291 webserver.cc:460] Webserver started at http://127.1.182.130:43067/ using document root <none> and password file <none>
I20260928 02:05:50.473554 12291 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:50.473663 12291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:50.473951 12291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:50.477211 12291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "6f6fbc69164e4de3a61d506149d19fff"
format_stamp: "Formatted at 2026-09-28 02:05:50 on dist-test-slave-23bp"
server_key: "02b68dd7e8601b1aac4ded14f38f2695"
server_key_iv: "97aff367340fc33493bc0e578f8f3f14"
server_key_version: "encryptionkey@0"
I20260928 02:05:50.478019 12291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "6f6fbc69164e4de3a61d506149d19fff"
format_stamp: "Formatted at 2026-09-28 02:05:50 on dist-test-slave-23bp"
server_key: "02b68dd7e8601b1aac4ded14f38f2695"
server_key_iv: "97aff367340fc33493bc0e578f8f3f14"
server_key_version: "encryptionkey@0"
I20260928 02:05:50.484004 12291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:05:50.487645 12306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:50.489158 12291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:05:50.489393 12291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "6f6fbc69164e4de3a61d506149d19fff"
format_stamp: "Formatted at 2026-09-28 02:05:50 on dist-test-slave-23bp"
server_key: "02b68dd7e8601b1aac4ded14f38f2695"
server_key_iv: "97aff367340fc33493bc0e578f8f3f14"
server_key_version: "encryptionkey@0"
I20260928 02:05:50.489583 12291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:50.503225 12291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:50.507648 12291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:50.507946 12291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:50.508890 12291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:50.510269 12291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:50.510340 12291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:50.510469 12291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:50.510521 12291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:50.546010 12291 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:46075
I20260928 02:05:50.546052 12420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:46075 every 8 connection(s)
I20260928 02:05:50.547389 12291 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:05:50.550829 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 12291
I20260928 02:05:50.551052 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:05:50.551730 1754 external_mini_cluster.cc:1428] Setting key 289ca7fdc24a31308667c73ed9a50cbf
Sep 28 02:05:50 dist-test-slave-23bp krb5kdc[12049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561150, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:05:50.569291 12287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:50.550241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:56935 (local address 127.1.182.190:36011)
0928 02:05:50.550425 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:50.550430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:50.551995 (+ 1565us) server_negotiation.cc:407] Connection header received
0928 02:05:50.553752 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:50.553757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:50.553837 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:50.553938 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:50.555623 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:50.556747 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:50.558431 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:50.559187 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:50.559324 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:50.563215 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:50.563221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:50.563224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:50.563253 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:50.565818 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:50.566736 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:50.566740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:50.566742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:50.566868 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:50.567355 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:50.567358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:50.567361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:50.567669 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:50.567778 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:50.568783 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0928 02:05:50.568972 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:05:50.570721 12423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:50.550979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:36011 (local address 127.1.182.130:56935)
0928 02:05:50.551781 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:50.551870 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:50.553485 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:50.554184 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:50.554195 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:50.554747 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:50.555443 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:50.555463 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:50.556907 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:50.556912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:50.558277 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:50.558290 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:50.559560 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:50.559567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:50.559784 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:50.560540 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:50.560563 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:50.562989 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:50.565965 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:50.565977 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:50.565997 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:50.566608 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:50.566973 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:50.566980 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:50.566987 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:50.567246 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:50.567799 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:50.567810 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:50.568290 (+ 480us) client_negotiation.cc:769] Sending connection context
0928 02:05:50.568640 (+ 350us) client_negotiation.cc:240] Negotiation successful
0928 02:05:50.569252 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":312,"threads_started":1}
I20260928 02:05:50.572388 12421 heartbeater.cc:347] Connected to a master server at 127.1.182.190:36011
I20260928 02:05:50.572718 12421 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:50.573473 12421 heartbeater.cc:511] Master 127.1.182.190:36011 requested a full tablet report, sending...
I20260928 02:05:50.575011 12094 ts_manager.cc:194] Registered new tserver with Master: 6f6fbc69164e4de3a61d506149d19fff (127.1.182.130:46075)
I20260928 02:05:50.576285 12094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:56935
WARNING: no policy specified for kudu/127.1.182.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
I20260928 02:05:50.605931 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:36011
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:05:50.933800 12426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:05:50.934063 12426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:05:50.934290 12426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:05:50.941562 12426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:05:50.941668 12426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:05:50.941892 12426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:05:50.951472 12426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:36011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:05:50.952764 12426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:05:50.954057 12426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:05:50.967496 12432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:50.967443 12434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:05:50.967983 12431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:05:50.968264 12426 server_base.cc:1034] running on GCE node
I20260928 02:05:50.968915 12426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:05:50.969859 12426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:05:50.971113 12426 hybrid_clock.cc:648] HybridClock initialized: now 1790561150971001 us; error 64 us; skew 500 ppm
I20260928 02:05:50.971454 12426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:05:50 dist-test-slave-23bp krb5kdc[12049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561150, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:05:50.974674 12426 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:05:50.976256 12426 webserver.cc:460] Webserver started at http://127.1.182.131:40477/ using document root <none> and password file <none>
I20260928 02:05:50.977017 12426 fs_manager.cc:359] Metadata directory not provided
I20260928 02:05:50.977161 12426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:05:50.977471 12426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:05:50.980633 12426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "1259a7adeb744c489117c238310fb601"
format_stamp: "Formatted at 2026-09-28 02:05:50 on dist-test-slave-23bp"
server_key: "97be209c22acf7e213e344ae20d07ede"
server_key_iv: "45263f72691f16de601e8c9302450f46"
server_key_version: "encryptionkey@0"
I20260928 02:05:50.981444 12426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "1259a7adeb744c489117c238310fb601"
format_stamp: "Formatted at 2026-09-28 02:05:50 on dist-test-slave-23bp"
server_key: "97be209c22acf7e213e344ae20d07ede"
server_key_iv: "45263f72691f16de601e8c9302450f46"
server_key_version: "encryptionkey@0"
I20260928 02:05:50.986701 12426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:05:50.990444 12441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:50.991852 12426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:05:50.992014 12426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "1259a7adeb744c489117c238310fb601"
format_stamp: "Formatted at 2026-09-28 02:05:50 on dist-test-slave-23bp"
server_key: "97be209c22acf7e213e344ae20d07ede"
server_key_iv: "45263f72691f16de601e8c9302450f46"
server_key_version: "encryptionkey@0"
I20260928 02:05:50.992228 12426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:05:51.025766 12426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:05:51.029709 12426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:05:51.030004 12426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:05:51.031018 12426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:05:51.032374 12426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:05:51.032446 12426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:51.032596 12426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:05:51.032648 12426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:05:51.065022 12426 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:38097
I20260928 02:05:51.065069 12553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:38097 every 8 connection(s)
I20260928 02:05:51.066315 12426 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:05:51.068455 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 12426
I20260928 02:05:51.068652 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:05:51.069273 1754 external_mini_cluster.cc:1428] Setting key bd940ab60886ddc839c96e840afa54f4
I20260928 02:05:51.074874 12285 heartbeater.cc:503] Master 127.1.182.190:36011 was elected leader, sending a full tablet report...
Sep 28 02:05:51 dist-test-slave-23bp krb5kdc[12049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561150, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:05:51.087446 12287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:51.069044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:48165 (local address 127.1.182.190:36011)
0928 02:05:51.069260 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:51.069266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:51.070711 (+ 1445us) server_negotiation.cc:407] Connection header received
0928 02:05:51.072973 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:51.072977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:51.073040 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:51.073120 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:51.074610 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.075662 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.077464 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.077949 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.078043 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:51.081435 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:51.081441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:51.081443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:51.081474 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:51.084337 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:51.085160 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:51.085168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:51.085174 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:51.085308 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:51.085717 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:51.085723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:51.085726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:51.086003 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:51.086155 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:51.086826 (+ 671us) server_negotiation.cc:299] Negotiation successful
0928 02:05:51.087047 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:05:51.088420 12556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:51.069730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:36011 (local address 127.1.182.131:48165)
0928 02:05:51.070520 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:51.070603 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:51.072707 (+ 2104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:51.073327 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:51.073356 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:51.073874 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:05:51.074436 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:51.074455 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.075827 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.075832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:51.077343 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:51.077354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.078131 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.078135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:51.078326 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:51.079021 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:05:51.079049 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:05:51.081240 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:05:51.084483 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:51.084494 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:51.084512 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:51.085044 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:51.085417 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:05:51.085422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:05:51.085426 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:05:51.085607 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:05:51.086131 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:05:51.086139 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:05:51.086491 (+ 352us) client_negotiation.cc:769] Sending connection context
0928 02:05:51.086766 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 02:05:51.087317 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":296,"threads_started":1}
I20260928 02:05:51.089797 12554 heartbeater.cc:347] Connected to a master server at 127.1.182.190:36011
I20260928 02:05:51.090106 12554 heartbeater.cc:464] Registering TS with master...
I20260928 02:05:51.090902 12554 heartbeater.cc:511] Master 127.1.182.190:36011 requested a full tablet report, sending...
I20260928 02:05:51.092371 12094 ts_manager.cc:194] Registered new tserver with Master: 1259a7adeb744c489117c238310fb601 (127.1.182.131:38097)
I20260928 02:05:51.093219 12094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:48165
I20260928 02:05:51.097289 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:05:51.114009 12287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:51.101663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54254 (local address 127.1.182.190:36011)
0928 02:05:51.101898 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:51.101907 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:51.102257 (+ 350us) server_negotiation.cc:407] Connection header received
0928 02:05:51.102409 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:51.102413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:51.102471 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:51.102551 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:51.103992 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.104701 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.105918 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.106300 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.106390 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:51.108791 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:51.108797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:51.108800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:51.108832 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:51.111462 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:51.112137 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:51.112140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:51.112143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:51.112274 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:51.112656 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:51.112659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:51.112661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:51.112990 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:51.113163 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:51.113503 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 02:05:51.113659 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:05:51.118278 12094 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:05:51.125216 12094 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:05:51.157846 12570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:51.137550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:35473 (local address 127.0.0.1:46778)
0928 02:05:51.148977 (+ 11427us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:51.149015 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:51.149194 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:51.150700 (+ 1506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:51.150707 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:51.150734 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:51.151257 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:51.151271 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.153200 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.153207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:51.155459 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:51.155474 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.156751 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.156756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:51.157025 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:51.157097 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 02:05:51.157222 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 02:05:51.157416 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11184,"mutex_wait_us":25,"thread_start_us":13539,"threads_started":1}
I20260928 02:05:51.157810 12569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:51.138049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46778 (local address 127.1.182.129:35473)
0928 02:05:51.148415 (+ 10366us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:51.148429 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:51.150150 (+ 1721us) server_negotiation.cc:407] Connection header received
0928 02:05:51.150265 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:51.150272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:51.150493 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:51.150699 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:51.151501 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.153057 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.155612 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.156556 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.156713 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:51.156969 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:51.157343 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 02:05:51.157502 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10151,"thread_start_us":9986,"threads_started":1}
I20260928 02:05:51.159723 12568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:51.137575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:38097 (local address 127.0.0.1:39138)
0928 02:05:51.151375 (+ 13800us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:51.151396 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:51.151515 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:51.152180 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:51.152184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:51.152206 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:51.152569 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:51.152582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.154490 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.154498 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:51.156864 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:51.156884 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.158889 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.158895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:51.159048 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:51.159116 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 02:05:51.159209 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 02:05:51.159335 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13661,"mutex_wait_us":22,"thread_start_us":11020,"threads_started":1}
I20260928 02:05:51.159749 12567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:51.137645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39138 (local address 127.1.182.131:38097)
0928 02:05:51.150187 (+ 12542us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:51.150196 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:51.151647 (+ 1451us) server_negotiation.cc:407] Connection header received
0928 02:05:51.151747 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:51.151754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:51.151997 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:51.152181 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:51.152819 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.154377 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.157016 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.158691 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.158834 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:51.159060 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:51.159244 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 02:05:51.159392 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12380,"thread_start_us":12250,"threads_started":1}
I20260928 02:05:51.161005 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:51.138435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:46075 (local address 127.0.0.1:34364)
0928 02:05:51.151785 (+ 13350us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:51.151800 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:51.151941 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:51.152560 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:51.152565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:51.152588 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:51.152949 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:51.152959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.154798 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.154806 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:51.157129 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:51.157141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.160248 (+ 3107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.160255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:51.160428 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:51.160459 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 02:05:51.160545 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:05:51.160660 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13221,"mutex_wait_us":72,"thread_start_us":13052,"threads_started":1}
I20260928 02:05:51.161654 12572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:51.139060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34364 (local address 127.1.182.130:46075)
0928 02:05:51.149678 (+ 10618us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:51.149690 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:51.151906 (+ 2216us) server_negotiation.cc:407] Connection header received
0928 02:05:51.152075 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:51.152083 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:51.152340 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:51.152541 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:51.153129 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.154657 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.157284 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.158691 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.159897 (+ 1206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:51.160092 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:51.161300 (+ 1208us) server_negotiation.cc:299] Negotiation successful
0928 02:05:51.161441 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10404,"thread_start_us":10269,"threads_started":1}
I20260928 02:05:51.162534 12219 tablet_service.cc:1467] Processing CreateTablet for tablet 2464b6242b794d31bfd5c1548a29fa6d (DEFAULT_TABLE table=test-table [id=14b0ece814ea4095b8c61169765111fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:05:51.164074 12219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2464b6242b794d31bfd5c1548a29fa6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:51.163956 12489 tablet_service.cc:1467] Processing CreateTablet for tablet 2464b6242b794d31bfd5c1548a29fa6d (DEFAULT_TABLE table=test-table [id=14b0ece814ea4095b8c61169765111fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:05:51.165301 12489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2464b6242b794d31bfd5c1548a29fa6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:51.165284 12354 tablet_service.cc:1467] Processing CreateTablet for tablet 2464b6242b794d31bfd5c1548a29fa6d (DEFAULT_TABLE table=test-table [id=14b0ece814ea4095b8c61169765111fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:05:51.166461 12354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2464b6242b794d31bfd5c1548a29fa6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:05:51.177336 12573 tablet_bootstrap.cc:492] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff: Bootstrap starting.
I20260928 02:05:51.179716 12574 tablet_bootstrap.cc:492] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d: Bootstrap starting.
I20260928 02:05:51.181223 12573 tablet_bootstrap.cc:654] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:51.182548 12573 log.cc:824] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff: Log is configured to *not* fsync() on all Append() calls
I20260928 02:05:51.183256 12574 tablet_bootstrap.cc:654] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:51.184047 12575 tablet_bootstrap.cc:492] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601: Bootstrap starting.
I20260928 02:05:51.184546 12574 log.cc:824] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:05:51.186077 12573 tablet_bootstrap.cc:492] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff: No bootstrap required, opened a new log
I20260928 02:05:51.186383 12573 ts_tablet_manager.cc:1397] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260928 02:05:51.188629 12575 tablet_bootstrap.cc:654] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601: Neither blocks nor log segments found. Creating new log.
I20260928 02:05:51.189644 12574 tablet_bootstrap.cc:492] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d: No bootstrap required, opened a new log
I20260928 02:05:51.189850 12575 log.cc:824] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601: Log is configured to *not* fsync() on all Append() calls
I20260928 02:05:51.189917 12574 ts_tablet_manager.cc:1397] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260928 02:05:51.192804 12575 tablet_bootstrap.cc:492] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601: No bootstrap required, opened a new log
I20260928 02:05:51.193078 12575 ts_tablet_manager.cc:1397] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260928 02:05:51.194756 12573 raft_consensus.cc:348] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1259a7adeb744c489117c238310fb601" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 38097 } } peers { permanent_uuid: "cc241e2c7f614c99a22370214eda562d" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35473 } } peers { permanent_uuid: "6f6fbc69164e4de3a61d506149d19fff" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 46075 } }
I20260928 02:05:51.195163 12573 raft_consensus.cc:374] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:51.195279 12573 raft_consensus.cc:729] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6fbc69164e4de3a61d506149d19fff, State: Initialized, Role: FOLLOWER
I20260928 02:05:51.195916 12573 consensus_queue.cc:260] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1259a7adeb744c489117c238310fb601" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 38097 } } peers { permanent_uuid: "cc241e2c7f614c99a22370214eda562d" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35473 } } peers { permanent_uuid: "6f6fbc69164e4de3a61d506149d19fff" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 46075 } }
I20260928 02:05:51.197129 12421 heartbeater.cc:503] Master 127.1.182.190:36011 was elected leader, sending a full tablet report...
I20260928 02:05:51.198978 12573 ts_tablet_manager.cc:1428] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20260928 02:05:51.200917 12574 raft_consensus.cc:348] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1259a7adeb744c489117c238310fb601" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 38097 } } peers { permanent_uuid: "cc241e2c7f614c99a22370214eda562d" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35473 } } peers { permanent_uuid: "6f6fbc69164e4de3a61d506149d19fff" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 46075 } }
I20260928 02:05:51.201363 12574 raft_consensus.cc:374] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:51.201486 12574 raft_consensus.cc:729] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc241e2c7f614c99a22370214eda562d, State: Initialized, Role: FOLLOWER
I20260928 02:05:51.202095 12574 consensus_queue.cc:260] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1259a7adeb744c489117c238310fb601" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 38097 } } peers { permanent_uuid: "cc241e2c7f614c99a22370214eda562d" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35473 } } peers { permanent_uuid: "6f6fbc69164e4de3a61d506149d19fff" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 46075 } }
I20260928 02:05:51.203847 12574 ts_tablet_manager.cc:1428] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260928 02:05:51.204222 12575 raft_consensus.cc:348] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1259a7adeb744c489117c238310fb601" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 38097 } } peers { permanent_uuid: "cc241e2c7f614c99a22370214eda562d" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35473 } } peers { permanent_uuid: "6f6fbc69164e4de3a61d506149d19fff" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 46075 } }
I20260928 02:05:51.204571 12575 raft_consensus.cc:374] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:05:51.204689 12575 raft_consensus.cc:729] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1259a7adeb744c489117c238310fb601, State: Initialized, Role: FOLLOWER
I20260928 02:05:51.205228 12575 consensus_queue.cc:260] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1259a7adeb744c489117c238310fb601" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 38097 } } peers { permanent_uuid: "cc241e2c7f614c99a22370214eda562d" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35473 } } peers { permanent_uuid: "6f6fbc69164e4de3a61d506149d19fff" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 46075 } }
I20260928 02:05:51.206241 12554 heartbeater.cc:503] Master 127.1.182.190:36011 was elected leader, sending a full tablet report...
I20260928 02:05:51.206974 12575 ts_tablet_manager.cc:1428] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
I20260928 02:05:51.268471 12580 raft_consensus.cc:482] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:05:51.268853 12580 raft_consensus.cc:504] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1259a7adeb744c489117c238310fb601" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 38097 } } peers { permanent_uuid: "cc241e2c7f614c99a22370214eda562d" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35473 } } peers { permanent_uuid: "6f6fbc69164e4de3a61d506149d19fff" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 46075 } }
I20260928 02:05:51.270812 12580 leader_election.cc:288] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1259a7adeb744c489117c238310fb601 (127.1.182.131:38097), 6f6fbc69164e4de3a61d506149d19fff (127.1.182.130:46075)
I20260928 02:05:51.277446 12572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:51.271488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:50015 (local address 127.1.182.130:46075)
0928 02:05:51.271684 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:51.271693 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:51.271868 (+ 175us) server_negotiation.cc:407] Connection header received
0928 02:05:51.272108 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:51.272116 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:51.272206 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:51.272301 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:51.272914 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.273688 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.275667 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.276436 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.276582 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:51.276673 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:51.277072 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 02:05:51.277189 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:05:51.277442 12567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:51.271278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:59023 (local address 127.1.182.131:38097)
0928 02:05:51.271811 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:51.271818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:51.272204 (+ 386us) server_negotiation.cc:407] Connection header received
0928 02:05:51.272340 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:51.272344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:51.272420 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:51.272531 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:05:51.273100 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.273866 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.275699 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.276462 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.276658 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:51.276727 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:51.277049 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 02:05:51.277168 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392}
I20260928 02:05:51.277751 12582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:51.271088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:46075 (local address 127.1.182.129:50015)
0928 02:05:51.271740 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:51.271765 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:51.271949 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:51.272326 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:51.272331 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:51.272357 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:51.272759 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:51.272772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.273842 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.273847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:51.275519 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:51.275531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.276666 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.276673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:51.276858 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:51.276920 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 02:05:51.277035 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 02:05:51.277160 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":44,"thread_start_us":319,"threads_started":1}
I20260928 02:05:51.277832 12583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:51.271361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:38097 (local address 127.1.182.129:59023)
0928 02:05:51.272094 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0928 02:05:51.272116 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:05:51.272233 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:05:51.272556 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:05:51.272559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:05:51.272577 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:05:51.272943 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:51.272955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.274003 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.274006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:51.275527 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:05:51.275539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.276712 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.276718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:05:51.276867 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:51.276921 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 02:05:51.276994 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:05:51.277134 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":54,"thread_start_us":443,"threads_started":1}
I20260928 02:05:51.278826 12374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2464b6242b794d31bfd5c1548a29fa6d" candidate_uuid: "cc241e2c7f614c99a22370214eda562d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f6fbc69164e4de3a61d506149d19fff" is_pre_election: true
I20260928 02:05:51.278972 12509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2464b6242b794d31bfd5c1548a29fa6d" candidate_uuid: "cc241e2c7f614c99a22370214eda562d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1259a7adeb744c489117c238310fb601" is_pre_election: true
I20260928 02:05:51.279491 12509 raft_consensus.cc:2454] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc241e2c7f614c99a22370214eda562d in term 0.
I20260928 02:05:51.279491 12374 raft_consensus.cc:2454] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc241e2c7f614c99a22370214eda562d in term 0.
I20260928 02:05:51.280156 12174 leader_election.cc:302] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f6fbc69164e4de3a61d506149d19fff, cc241e2c7f614c99a22370214eda562d; no voters:
I20260928 02:05:51.280488 12580 raft_consensus.cc:2781] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:05:51.280651 12580 raft_consensus.cc:482] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:05:51.280778 12580 raft_consensus.cc:3037] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:51.283061 12580 raft_consensus.cc:504] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1259a7adeb744c489117c238310fb601" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 38097 } } peers { permanent_uuid: "cc241e2c7f614c99a22370214eda562d" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35473 } } peers { permanent_uuid: "6f6fbc69164e4de3a61d506149d19fff" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 46075 } }
I20260928 02:05:51.283816 12580 leader_election.cc:288] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d [CANDIDATE]: Term 1 election: Requested vote from peers 1259a7adeb744c489117c238310fb601 (127.1.182.131:38097), 6f6fbc69164e4de3a61d506149d19fff (127.1.182.130:46075)
I20260928 02:05:51.284268 12509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2464b6242b794d31bfd5c1548a29fa6d" candidate_uuid: "cc241e2c7f614c99a22370214eda562d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1259a7adeb744c489117c238310fb601"
I20260928 02:05:51.284338 12374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2464b6242b794d31bfd5c1548a29fa6d" candidate_uuid: "cc241e2c7f614c99a22370214eda562d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f6fbc69164e4de3a61d506149d19fff"
I20260928 02:05:51.284514 12509 raft_consensus.cc:3037] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:51.284543 12374 raft_consensus.cc:3037] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:05:51.286674 12509 raft_consensus.cc:2454] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc241e2c7f614c99a22370214eda562d in term 1.
I20260928 02:05:51.286696 12374 raft_consensus.cc:2454] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc241e2c7f614c99a22370214eda562d in term 1.
I20260928 02:05:51.287138 12175 leader_election.cc:302] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1259a7adeb744c489117c238310fb601, cc241e2c7f614c99a22370214eda562d; no voters:
I20260928 02:05:51.287439 12580 raft_consensus.cc:2781] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:05:51.287825 12580 raft_consensus.cc:686] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d [term 1 LEADER]: Becoming Leader. State: Replica: cc241e2c7f614c99a22370214eda562d, State: Running, Role: LEADER
I20260928 02:05:51.288383 12580 consensus_queue.cc:237] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1259a7adeb744c489117c238310fb601" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 38097 } } peers { permanent_uuid: "cc241e2c7f614c99a22370214eda562d" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35473 } } peers { permanent_uuid: "6f6fbc69164e4de3a61d506149d19fff" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 46075 } }
I20260928 02:05:51.293030 12094 catalog_manager.cc:5495] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d reported cstate change: term changed from 0 to 1, leader changed from <none> to cc241e2c7f614c99a22370214eda562d (127.1.182.129). New cstate: current_term: 1 leader_uuid: "cc241e2c7f614c99a22370214eda562d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1259a7adeb744c489117c238310fb601" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 38097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc241e2c7f614c99a22370214eda562d" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 35473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f6fbc69164e4de3a61d506149d19fff" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 46075 } health_report { overall_health: UNKNOWN } } }
W20260928 02:05:51.296025 12286 tablet.cc:2369] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:05:51.299268 12422 tablet.cc:2369] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:05:51.318028 12555 tablet.cc:2369] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:05:51.339023 12569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:51.333598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46788 (local address 127.1.182.129:35473)
0928 02:05:51.333816 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:51.333824 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:51.333919 (+ 95us) server_negotiation.cc:407] Connection header received
0928 02:05:51.334165 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:51.334169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:51.334309 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:51.334462 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:05:51.334844 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.335638 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.336962 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.337378 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.337506 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:51.337974 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:05:51.338470 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:05:51.338636 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:51.338684 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 02:05:51.338763 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:05:51.347813 12509 raft_consensus.cc:1261] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601 [term 1 FOLLOWER]: Refusing update from remote peer cc241e2c7f614c99a22370214eda562d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:05:51.347820 12374 raft_consensus.cc:1261] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff [term 1 FOLLOWER]: Refusing update from remote peer cc241e2c7f614c99a22370214eda562d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:05:51.348799 12580 consensus_queue.cc:1035] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1259a7adeb744c489117c238310fb601" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 38097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:05:51.349083 12584 consensus_queue.cc:1035] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f6fbc69164e4de3a61d506149d19fff" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 46075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:05:51.361871 12588 mvcc.cc:205] Tried to move back new op lower bound from 7334138475910111232 to 7334138475681460224. Current Snapshot: MvccSnapshot[applied={T|T < 7334138475910111232}]
I20260928 02:05:51.376251 12093 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:05:51.376602 12093 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:05:51.380999 12093 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c3bca5e50cb4aeea01c5d650fd7f509: Sending DeleteTablet for 3 replicas of tablet 2464b6242b794d31bfd5c1548a29fa6d
I20260928 02:05:51.382259 12489 tablet_service.cc:1514] Processing DeleteTablet for tablet 2464b6242b794d31bfd5c1548a29fa6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:05:51 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:39138
I20260928 02:05:51.382421 12354 tablet_service.cc:1514] Processing DeleteTablet for tablet 2464b6242b794d31bfd5c1548a29fa6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:05:51 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:34364
I20260928 02:05:51.382439 12219 tablet_service.cc:1514] Processing DeleteTablet for tablet 2464b6242b794d31bfd5c1548a29fa6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:05:51 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:46778
I20260928 02:05:51.386726 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 12155
I20260928 02:05:51.395889 12599 tablet_replica.cc:330] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d: stopping tablet replica
I20260928 02:05:51.396556 12599 raft_consensus.cc:2229] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:05:51.397349 12599 raft_consensus.cc:2258] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:05:51.399718 12599 ts_tablet_manager.cc:1905] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:51.408046 12599 ts_tablet_manager.cc:1918] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:05:51.408236 12599 log.cc:1196] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-0/wal/wals/2464b6242b794d31bfd5c1548a29fa6d
I20260928 02:05:51.408891 12599 ts_tablet_manager.cc:1939] T 2464b6242b794d31bfd5c1548a29fa6d P cc241e2c7f614c99a22370214eda562d: Deleting consensus metadata
I20260928 02:05:51.410964 12079 catalog_manager.cc:4841] TS cc241e2c7f614c99a22370214eda562d (127.1.182.129:35473): tablet 2464b6242b794d31bfd5c1548a29fa6d (table test-table [id=14b0ece814ea4095b8c61169765111fa]) successfully deleted
I20260928 02:05:51.416633 12598 tablet_replica.cc:330] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff: stopping tablet replica
I20260928 02:05:51.417176 12598 raft_consensus.cc:2229] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:05:51.417740 12598 raft_consensus.cc:2258] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 28 02:05:51 dist-test-slave-23bp krb5kdc[12049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:05:51.421300 12597 tablet_replica.cc:330] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601: stopping tablet replica
I20260928 02:05:51.421744 12597 raft_consensus.cc:2229] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:05:51.422274 12597 raft_consensus.cc:2258] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:05:51.420348 12598 ts_tablet_manager.cc:1905] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:51.428442 12597 ts_tablet_manager.cc:1905] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:05:51.430655 12598 ts_tablet_manager.cc:1918] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:05:51.430902 12598 log.cc:1196] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/2464b6242b794d31bfd5c1548a29fa6d
I20260928 02:05:51.431607 12598 ts_tablet_manager.cc:1939] T 2464b6242b794d31bfd5c1548a29fa6d P 6f6fbc69164e4de3a61d506149d19fff: Deleting consensus metadata
I20260928 02:05:51.433678 12080 catalog_manager.cc:4841] TS 6f6fbc69164e4de3a61d506149d19fff (127.1.182.130:46075): tablet 2464b6242b794d31bfd5c1548a29fa6d (table test-table [id=14b0ece814ea4095b8c61169765111fa]) successfully deleted
I20260928 02:05:51.435045 12597 ts_tablet_manager.cc:1918] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:05:51.435231 12597 log.cc:1196] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/2464b6242b794d31bfd5c1548a29fa6d
I20260928 02:05:51.435721 12597 ts_tablet_manager.cc:1939] T 2464b6242b794d31bfd5c1548a29fa6d P 1259a7adeb744c489117c238310fb601: Deleting consensus metadata
I20260928 02:05:51.437296 12081 catalog_manager.cc:4841] TS 1259a7adeb744c489117c238310fb601 (127.1.182.131:38097): tablet 2464b6242b794d31bfd5c1548a29fa6d (table test-table [id=14b0ece814ea4095b8c61169765111fa]) successfully deleted
I20260928 02:05:51.437821 12569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:51.388427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46802 (local address 127.1.182.129:35473)
0928 02:05:51.388684 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:51.388693 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:51.407597 (+ 18904us) server_negotiation.cc:407] Connection header received
0928 02:05:51.408785 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:51.408794 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:51.408909 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:51.408989 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:51.411665 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.413282 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.415896 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.416449 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.416523 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:51.421755 (+ 5232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:51.421784 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:51.421803 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:51.421854 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:51.426155 (+ 4301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:51.434347 (+ 8192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:51.434360 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:51.434364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:51.434609 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:51.435418 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:51.435424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:51.435429 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:51.435968 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:51.436284 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:51.437053 (+ 769us) server_negotiation.cc:299] Negotiation successful
0928 02:05:51.437338 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:05:51.440505 12280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:51.540781 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 12155
I20260928 02:05:51.559423 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 12291
Sep 28 02:05:51 dist-test-slave-23bp krb5kdc[12049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.130@KRBTEST.COM
I20260928 02:05:51.574779 12572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:51.560185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34376 (local address 127.1.182.130:46075)
0928 02:05:51.560379 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:51.560383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:51.560452 (+ 69us) server_negotiation.cc:407] Connection header received
0928 02:05:51.560645 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:51.560648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:51.560716 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:51.560793 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:51.562392 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.563533 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.564921 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.565313 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.565414 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:51.569392 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:51.569403 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:51.569418 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:51.569450 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:51.572336 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:51.572983 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:51.572989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:51.572991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:51.573107 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:51.573423 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:51.573427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:51.573429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:51.573775 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:51.573992 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:51.574231 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 02:05:51.574388 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 02:05:51.575469 12416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:51.664709 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 12291
I20260928 02:05:51.681528 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 12426
Sep 28 02:05:51 dist-test-slave-23bp krb5kdc[12049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.131@KRBTEST.COM
I20260928 02:05:51.697206 12567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:05:51.682325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39154 (local address 127.1.182.131:38097)
0928 02:05:51.682514 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:05:51.682519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:05:51.682536 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:05:51.682648 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:05:51.682652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:05:51.682710 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:05:51.682802 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:05:51.684074 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.684905 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.686116 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:05:51.686546 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:05:51.686653 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:05:51.691384 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:05:51.691399 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:05:51.691416 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:05:51.691453 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:05:51.694201 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:51.694934 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:51.694943 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:51.694945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:51.695136 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:05:51.695569 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:05:51.695577 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:05:51.695580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:05:51.695954 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:05:51.696275 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 02:05:51.696624 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 02:05:51.696871 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 02:05:51.697855 12549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:51.785343 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 12426
I20260928 02:05:51.802002 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 12062
I20260928 02:05:51.803228 12124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:05:51.915302 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 12062
2026-09-28T02:05:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4346 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:05:52 dist-test-slave-23bp krb5kdc[12607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:05:52 dist-test-slave-23bp krb5kdc[12607](info): set up 2 sockets
Sep 28 02:05:52 dist-test-slave-23bp krb5kdc[12607](info): commencing operation
krb5kdc: starting...
W20260928 02:05:53.026525 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:05:53 dist-test-slave-23bp krb5kdc[12607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:05:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:05:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.182.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.182.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.182.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/rangeradmin_127.1.182.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.182.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/rangeradmin_127.1.182.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.182.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.182.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.182.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/rangerlookup_127.1.182.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.182.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/rangerlookup_127.1.182.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.182.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.182.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.182.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/HTTP_127.1.182.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.182.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/HTTP_127.1.182.148@KRBTEST.COM.keytab.
I20260928 02:05:53.354384 1754 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasktsFKfV/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/postgres -l logfile start
2026-09-28 02:05:55.906 UTC [12638] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 02:05:55.907 UTC [12638] LOG: listening on IPv4 address "127.1.182.148", port 38715
2026-09-28 02:05:55.910 UTC [12638] LOG: listening on Unix socket "/tmp/.s.PGSQL.38715"
2026-09-28 02:05:55.931 UTC [12641] LOG: database system was shut down at 2026-09-28 02:05:54 UTC
2026-09-28 02:05:55.935 UTC [12638] LOG: database system is ready to accept connections
I20260928 02:05:56.910115 1754 mini_postgres.cc:96] Postgres bound to 38715
2026-09-28 02:05:56.928 UTC [12649] FATAL: database "slave" does not exist
127.1.182.148:38715 - accepting connections
I20260928 02:05:56.928778 1754 mini_ranger.cc:161] Starting Ranger...
I20260928 02:05:56.955902 1754 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 02:05:57.441048 1754 mini_ranger.cc:87] Created ranger Postgres database
I20260928 02:05:57.441177 1754 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ranger-admin
2026-09-28 02:05:57,718 [I] DB FLAVOR :POSTGRES
2026-09-28 02:05:57,718 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:05:57,718 [I] Checking connection..
2026-09-28 02:05:57,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 02:05:57,973 [I] Checking connection passed.
2026-09-28 02:05:57,973 [I] --------- Verifying version history table ---------
2026-09-28 02:05:57,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:05:58,254 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:05:58,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:05:58,511 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:05:58,511 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 02:05:58,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 02:05:58.751 UTC [12714] WARNING: there is no transaction in progress
2026-09-28 02:05:58,757 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 02:05:58,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:05:59,002 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 02:05:59,002 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:05:59,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 02:05:59,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 02:05:59,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-23bp.c.gcp-upstream.internal','N') ;"
2026-09-28 02:05:59,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:06:00,007 [I] Table x_portal_user does not exist in database ranger
2026-09-28 02:06:00,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:06:00,270 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 02:06:00,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:06:00,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:06:00,749 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 02:06:00,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 02:06:01.748 UTC [12822] WARNING: there is no transaction in progress
2026-09-28 02:06:01.766 UTC [12822] WARNING: there is no transaction in progress
2026-09-28 02:06:01.783 UTC [12822] WARNING: there is no transaction in progress
2026-09-28 02:06:01.799 UTC [12822] WARNING: there is no transaction in progress
2026-09-28 02:06:01.815 UTC [12822] WARNING: there is no transaction in progress
2026-09-28 02:06:01.833 UTC [12822] WARNING: there is no transaction in progress
2026-09-28 02:06:01.891 UTC [12822] WARNING: there is no transaction in progress
2026-09-28 02:06:01.898 UTC [12822] WARNING: there is no transaction in progress
2026-09-28 02:06:01.913 UTC [12822] WARNING: there is no transaction in progress
2026-09-28 02:06:01.924 UTC [12822] WARNING: there is no transaction in progress
2026-09-28 02:06:01.934 UTC [12822] WARNING: there is no transaction in progress
2026-09-28 02:06:01.948 UTC [12822] WARNING: there is no transaction in progress
2026-09-28 02:06:01.958 UTC [12822] WARNING: there is no transaction in progress
2026-09-28 02:06:01.965 UTC [12822] WARNING: there is no transaction in progress
2026-09-28 02:06:01.973 UTC [12822] WARNING: there is no transaction in progress
2026-09-28 02:06:01.980 UTC [12822] WARNING: there is no transaction in progress
2026-09-28 02:06:02,633 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 02:06:02,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 02:06:02,893 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 02:06:02,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:06:03,140 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 02:06:03,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:06:03,396 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 02:06:03,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:06:03,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:06:03,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-23bp.c.gcp-upstream.internal';"
2026-09-28 02:06:04,131 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 02:06:04,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 02:06:04,382 [I] DB_PATCHES have already been applied
I20260928 02:06:04.387904 1754 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ranger-admin:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasktsFKfV/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 02:06:04.388013 1754 mini_ranger.cc:193] Using host: 127.1.182.148
I20260928 02:06:04.403013 1754 mini_ranger.cc:227] Ranger admin URL: http://127.1.182.148:57971
Sep 28, 2026 2:06:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:06:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:06:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:06:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:06:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:06:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:06:05 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 02:06:05.577617 1754 mini_ranger.cc:160] Time spent starting Ranger: real 8.649s user 0.005s sys 0.034s
Sep 28, 2026 2:06:24 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:06:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:06:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:06:24 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:06:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:06:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:06:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:06:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:06:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:06:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:06:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:06:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:06:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:06:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:06:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:06:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:06:24 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:06:25 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 02:06:26.037098 1754 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.182.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.182.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.182.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/rangerkms_127.1.182.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.182.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/rangerkms_127.1.182.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.182.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.182.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.182.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/HTTP_127.1.182.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.182.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/HTTP_127.1.182.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 02:06:26 dist-test-slave-23bp krb5kdc[12607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561186, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 02:06:26.229738 1754 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 02:06:26.251341 1754 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 02:06:27.136933 1754 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 02:06:27.137068 1754 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ranger-kms
I20260928 02:06:27.137238 1754 mini_ranger_kms.cc:227] Using postgres at 127.1.182.148:38715
2026-09-28 02:06:27,556 [I] DB FLAVOR :POSTGRES
2026-09-28 02:06:27,556 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:06:27,556 [I] Checking connection
2026-09-28 02:06:27,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 02:06:27,815 [I] connection success
2026-09-28 02:06:27,816 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 02:06:27,816 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 02:06:27,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 02:06:28,055 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 02:06:28,056 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:06:28,056 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 02:06:28,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:38715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 02:06:28,322 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 02:06:28.778322 1754 mini_ranger_kms.cc:310] Created kms service
I20260928 02:06:28.969810 1754 mini_ranger_kms.cc:326] Created kudu user
I20260928 02:06:29.012635 1754 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 02:06:29 dist-test-slave-23bp krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561186, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
I20260928 02:06:29.407014 1754 mini_ranger_kms.cc:384] Added ranger policy
I20260928 02:06:29.407379 1754 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ranger-kms:/tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 02:06:29.407449 1754 mini_ranger_kms.cc:240] Using host: 127.1.182.148
I20260928 02:06:29.425141 1754 mini_ranger_kms.cc:276] Ranger KMS PID: 13004
I20260928 02:06:29.425374 1754 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.182.148:59189
Sep 28, 2026 2:06:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:06:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:06:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:06:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:06:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:06:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 02:06:31.710773 1754 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.481s user 0.104s sys 0.334s
I20260928 02:06:31.711024 1754 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 02:06:31.712109 1754 mini_ranger_kms.cc:401] 127.1.182.148:59189/kms/v1/keys
Sep 28 02:06:31 dist-test-slave-23bp krb5kdc[12607](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790561191, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.182.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 02:06:32 dist-test-slave-23bp krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561191, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.182.148@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
Sep 28 02:06:32 dist-test-slave-23bp krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561191, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.182.148@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
Sep 28 02:06:34 dist-test-slave-23bp krb5kdc[12607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:06:34.815531 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:39053
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:39053
--ranger_config_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.182.148:59189/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:06:35.165102 13055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:06:35.165463 13055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:06:35.165632 13055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:06:35.173243 13055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:06:35.173390 13055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:06:35.173491 13055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:06:35.173590 13055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:06:35.173678 13055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:06:35.183712 13055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.182.148:59189/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:39053
--ranger_config_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:39053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:06:35.185017 13055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:06:35.186333 13055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:06:35.198747 13060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:06:35.198711 13063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:06:35.198762 13061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:06:35.199491 13055 server_base.cc:1034] running on GCE node
I20260928 02:06:35.200179 13055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:06:35.201520 13055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:06:35.202716 13055 hybrid_clock.cc:648] HybridClock initialized: now 1790561195202681 us; error 33 us; skew 500 ppm
I20260928 02:06:35.203018 13055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:06:35 dist-test-slave-23bp krb5kdc[12607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561195, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:06:35.206096 13055 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:06:35.207624 13055 webserver.cc:460] Webserver started at http://127.1.182.190:42501/ using document root <none> and password file <none>
I20260928 02:06:35.208680 13055 fs_manager.cc:359] Metadata directory not provided
I20260928 02:06:35.208809 13055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:06:35.209121 13055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:06:35 dist-test-slave-23bp krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561195, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
I20260928 02:06:35.265558 13055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "038426ad89b54a0cac58d233a14e07e7"
format_stamp: "Formatted at 2026-09-28 02:06:35 on dist-test-slave-23bp"
server_key: "2b0c9e1fbd0055bc1d68bcc1ef1e930e"
server_key_iv: "79b374140c47b3bd36a5a7ab7a805cfa"
server_key_version: "kuduclusterkey@0"
I20260928 02:06:35.266613 13055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "038426ad89b54a0cac58d233a14e07e7"
format_stamp: "Formatted at 2026-09-28 02:06:35 on dist-test-slave-23bp"
server_key: "2b0c9e1fbd0055bc1d68bcc1ef1e930e"
server_key_iv: "79b374140c47b3bd36a5a7ab7a805cfa"
server_key_version: "kuduclusterkey@0"
I20260928 02:06:35.274016 13055 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260928 02:06:35.295849 13070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:06:35.297948 13055 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260928 02:06:35.298159 13055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "038426ad89b54a0cac58d233a14e07e7"
format_stamp: "Formatted at 2026-09-28 02:06:35 on dist-test-slave-23bp"
server_key: "2b0c9e1fbd0055bc1d68bcc1ef1e930e"
server_key_iv: "79b374140c47b3bd36a5a7ab7a805cfa"
server_key_version: "kuduclusterkey@0"
I20260928 02:06:35.298439 13055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:06:35.324645 13055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:06:35.328262 13055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:06:35.328583 13055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:06:35.348340 13055 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:39053
I20260928 02:06:35.348335 13121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:39053 every 8 connection(s)
I20260928 02:06:35.349736 13055 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:06:35.353540 13122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:06:35.359396 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 13055
I20260928 02:06:35.359603 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:06:35.360231 1754 external_mini_cluster.cc:1428] Setting key 0126b435972a7f96374296ebc534b924
I20260928 02:06:35.364933 13122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7: Bootstrap starting.
I20260928 02:06:35.368785 13122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7: Neither blocks nor log segments found. Creating new log.
I20260928 02:06:35.370015 13122 log.cc:824] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:06:35.373476 13122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7: No bootstrap required, opened a new log
Sep 28 02:06:35 dist-test-slave-23bp krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:06:35.381531 13122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "038426ad89b54a0cac58d233a14e07e7" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39053 } }
I20260928 02:06:35.381915 13122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:06:35.382045 13122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038426ad89b54a0cac58d233a14e07e7, State: Initialized, Role: FOLLOWER
I20260928 02:06:35.382659 13122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "038426ad89b54a0cac58d233a14e07e7" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39053 } }
I20260928 02:06:35.382937 13122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:06:35.383165 13122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:06:35.383348 13122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:06:35.383582 13125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:35.362068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57802 (local address 127.1.182.190:39053)
0928 02:06:35.362926 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0928 02:06:35.362937 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:06:35.362983 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:06:35.364652 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:06:35.364675 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:06:35.365120 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:06:35.365471 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:06:35.367188 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:35.368499 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:35.370266 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:35.371334 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:35.371494 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:35.375885 (+ 4391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:06:35.375892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:06:35.375909 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:06:35.375940 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:06:35.379363 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:06:35.380064 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:06:35.380074 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:06:35.380087 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:06:35.380229 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:06:35.380642 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:06:35.380647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:06:35.380649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:06:35.381146 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:06:35.381435 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 02:06:35.381896 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 02:06:35.382285 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":416,"threads_started":1}
I20260928 02:06:35.386428 13122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "038426ad89b54a0cac58d233a14e07e7" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39053 } }
I20260928 02:06:35.386929 13122 leader_election.cc:302] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 038426ad89b54a0cac58d233a14e07e7; no voters:
I20260928 02:06:35.387521 13122 leader_election.cc:288] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:06:35.388062 13127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:06:35.388741 13127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7 [term 1 LEADER]: Becoming Leader. State: Replica: 038426ad89b54a0cac58d233a14e07e7, State: Running, Role: LEADER
I20260928 02:06:35.389276 13127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "038426ad89b54a0cac58d233a14e07e7" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39053 } }
I20260928 02:06:35.389920 13122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:06:35.392514 13129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 038426ad89b54a0cac58d233a14e07e7. Latest consensus state: current_term: 1 leader_uuid: "038426ad89b54a0cac58d233a14e07e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "038426ad89b54a0cac58d233a14e07e7" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39053 } } }
I20260928 02:06:35.392882 13129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:06:35.392853 13128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "038426ad89b54a0cac58d233a14e07e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "038426ad89b54a0cac58d233a14e07e7" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39053 } } }
I20260928 02:06:35.393285 13128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:06:35.397403 13133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:06:35.403052 13133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:06:35.403750 13122 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:06:35.436029 13133 catalog_manager.cc:1347] Generated new cluster ID: 6f326ea5b4464f22953f76b2f57a69e5
I20260928 02:06:35.436220 13133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:06:35.447291 13133 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:06:35.449378 13133 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:06:35.472889 13133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7: Generated new TSK 0
I20260928 02:06:35.474042 13133 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.182.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
I20260928 02:06:36.247737 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.182.148:59189/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39053
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:06:36 dist-test-slave-23bp krb5kdc[12607](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790561196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 02:06:36.666286 13167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:06:36.666639 13167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:06:36.666832 13167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:06:36.675669 13167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:06:36.675822 13167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:06:36.676028 13167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:06:36.687705 13167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.182.148:59189/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:06:36.689332 13167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:06:36.690992 13167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:06:36.707379 13177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:06:36.707428 13178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:06:36.708222 13180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:06:36.709661 13167 server_base.cc:1034] running on GCE node
I20260928 02:06:36.710325 13167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:06:36.711555 13167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:06:36.712841 13167 hybrid_clock.cc:648] HybridClock initialized: now 1790561196712780 us; error 59 us; skew 500 ppm
I20260928 02:06:36.713181 13167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:06:36 dist-test-slave-23bp krb5kdc[12607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:06:36.717831 13167 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:06:36.720482 13167 webserver.cc:460] Webserver started at http://127.1.182.129:45797/ using document root <none> and password file <none>
I20260928 02:06:36.721230 13167 fs_manager.cc:359] Metadata directory not provided
I20260928 02:06:36.721369 13167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:06:36.721675 13167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:06:36 dist-test-slave-23bp krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
I20260928 02:06:36.738436 13167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "fa9acde1b66e4e7cb200c97145d4e572"
format_stamp: "Formatted at 2026-09-28 02:06:36 on dist-test-slave-23bp"
server_key: "28c9f3ef121b67a70f12ce8681e74628"
server_key_iv: "3e989a30bf9c6e302e87acae4a4595aa"
server_key_version: "kuduclusterkey@0"
I20260928 02:06:36.739624 13167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "fa9acde1b66e4e7cb200c97145d4e572"
format_stamp: "Formatted at 2026-09-28 02:06:36 on dist-test-slave-23bp"
server_key: "28c9f3ef121b67a70f12ce8681e74628"
server_key_iv: "3e989a30bf9c6e302e87acae4a4595aa"
server_key_version: "kuduclusterkey@0"
I20260928 02:06:36.753293 13167 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.008s
I20260928 02:06:36.791791 13188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:06:36.793946 13167 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.013s sys 0.007s
I20260928 02:06:36.794242 13167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "fa9acde1b66e4e7cb200c97145d4e572"
format_stamp: "Formatted at 2026-09-28 02:06:36 on dist-test-slave-23bp"
server_key: "28c9f3ef121b67a70f12ce8681e74628"
server_key_iv: "3e989a30bf9c6e302e87acae4a4595aa"
server_key_version: "kuduclusterkey@0"
I20260928 02:06:36.794664 13167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:06:36.822494 13167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:06:36.827627 13167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:06:36.827980 13167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:06:36.829182 13167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:06:36.837244 13167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:06:36.837353 13167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:06:36.837491 13167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:06:36.837549 13167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:06:36.899610 13300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:44035 every 8 connection(s)
I20260928 02:06:36.900169 13167 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:44035
I20260928 02:06:36.907232 13167 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:06:36.910037 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 13167
I20260928 02:06:36.910247 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:06:36.911041 1754 external_mini_cluster.cc:1428] Setting key 02e3d9c538314d8d2538e4acabcd6c02
Sep 28 02:06:36 dist-test-slave-23bp krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.182.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.130@KRBTEST.COM" created.
I20260928 02:06:36.975242 13304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:36.913500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:52823 (local address 127.1.182.190:39053)
0928 02:06:36.938285 (+ 24785us) server_negotiation.cc:206] Beginning negotiation
0928 02:06:36.938292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:06:36.942515 (+ 4223us) server_negotiation.cc:407] Connection header received
0928 02:06:36.942593 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:06:36.942612 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:06:36.942716 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:06:36.942840 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:06:36.949400 (+ 6560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:36.950529 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:36.957908 (+ 7379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:36.958670 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:36.958808 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:36.967624 (+ 8816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:06:36.967630 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:06:36.967634 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:06:36.967697 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:06:36.971179 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:06:36.972213 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:06:36.972219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:06:36.972222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:06:36.972371 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:06:36.972938 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:06:36.972944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:06:36.972947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:06:36.973384 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:06:36.973518 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 02:06:36.974579 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0928 02:06:36.974810 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24622,"thread_start_us":24801,"threads_started":1}
I20260928 02:06:36.977895 13303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:36.912991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39053 (local address 127.1.182.129:52823)
0928 02:06:36.939110 (+ 26119us) negotiation.cc:107] Waiting for socket to connect
0928 02:06:36.939186 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0928 02:06:36.942018 (+ 2832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:06:36.947453 (+ 5435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:06:36.947468 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:06:36.948213 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:06:36.949158 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:06:36.949186 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:06:36.950771 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:06:36.950776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:06:36.957684 (+ 6908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:06:36.957708 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:06:36.959351 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:06:36.959360 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:06:36.959698 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:36.962294 (+ 2596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:06:36.962338 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:06:36.967170 (+ 4832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:06:36.971371 (+ 4201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:06:36.971382 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:06:36.971401 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:06:36.972038 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:06:36.972542 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:06:36.972547 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:06:36.972551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:06:36.972802 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:06:36.973615 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:06:36.973624 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:06:36.974127 (+ 503us) client_negotiation.cc:769] Sending connection context
0928 02:06:36.975810 (+ 1683us) client_negotiation.cc:240] Negotiation successful
0928 02:06:36.976413 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25702,"thread_start_us":29861,"threads_started":1}
I20260928 02:06:36.980046 13301 heartbeater.cc:347] Connected to a master server at 127.1.182.190:39053
I20260928 02:06:36.980381 13301 heartbeater.cc:464] Registering TS with master...
I20260928 02:06:36.981212 13301 heartbeater.cc:511] Master 127.1.182.190:39053 requested a full tablet report, sending...
I20260928 02:06:36.983812 13087 ts_manager.cc:194] Registered new tserver with Master: fa9acde1b66e4e7cb200c97145d4e572 (127.1.182.129:44035)
I20260928 02:06:36.985854 13087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:52823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
I20260928 02:06:37.006880 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.182.148:59189/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39053
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:06:37 dist-test-slave-23bp krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
Sep 28 02:06:37 dist-test-slave-23bp krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
W20260928 02:06:37.508221 13307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:06:37.508839 13307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:06:37.509060 13307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:06:37.520781 13307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:06:37.521009 13307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:06:37.521261 13307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:06:37.536852 13307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.182.148:59189/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:06:37.538902 13307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:06:37.540717 13307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:06:37.587397 13314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:06:37.591504 13315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:06:37.595384 13318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:06:37.596014 13307 server_base.cc:1034] running on GCE node
I20260928 02:06:37.596980 13307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:06:37.603130 13307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:06:37.604508 13307 hybrid_clock.cc:648] HybridClock initialized: now 1790561197604424 us; error 70 us; skew 500 ppm
I20260928 02:06:37.604976 13307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:06:37 dist-test-slave-23bp krb5kdc[12607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:06:37.609333 13307 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:06:37.626636 13307 webserver.cc:460] Webserver started at http://127.1.182.130:43127/ using document root <none> and password file <none>
I20260928 02:06:37.627717 13307 fs_manager.cc:359] Metadata directory not provided
I20260928 02:06:37.627867 13307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:06:37.628235 13307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:06:37 dist-test-slave-23bp krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
I20260928 02:06:37.664367 13307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "92adcc0a73274c0aa8d163213996f4a6"
format_stamp: "Formatted at 2026-09-28 02:06:37 on dist-test-slave-23bp"
server_key: "67179c5ca781437746f073274d2854f9"
server_key_iv: "dc7439eaf7a8f88ffebcf2880aa6e57b"
server_key_version: "kuduclusterkey@0"
I20260928 02:06:37.665680 13307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "92adcc0a73274c0aa8d163213996f4a6"
format_stamp: "Formatted at 2026-09-28 02:06:37 on dist-test-slave-23bp"
server_key: "67179c5ca781437746f073274d2854f9"
server_key_iv: "dc7439eaf7a8f88ffebcf2880aa6e57b"
server_key_version: "kuduclusterkey@0"
I20260928 02:06:37.673309 13307 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260928 02:06:37.702829 13324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:06:37.704879 13307 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.005s
I20260928 02:06:37.705080 13307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "92adcc0a73274c0aa8d163213996f4a6"
format_stamp: "Formatted at 2026-09-28 02:06:37 on dist-test-slave-23bp"
server_key: "67179c5ca781437746f073274d2854f9"
server_key_iv: "dc7439eaf7a8f88ffebcf2880aa6e57b"
server_key_version: "kuduclusterkey@0"
I20260928 02:06:37.705374 13307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:06:37.747247 13307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:06:37.752058 13307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:06:37.752406 13307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:06:37.753556 13307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:06:37.755327 13307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:06:37.755420 13307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:06:37.755541 13307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:06:37.755604 13307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:06:37.801280 13307 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:33109
I20260928 02:06:37.801340 13445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:33109 every 8 connection(s)
I20260928 02:06:37.802933 13307 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:06:37.804471 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 13307
I20260928 02:06:37.804634 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:06:37.805234 1754 external_mini_cluster.cc:1428] Setting key 4d3db6768dab695d6cda590d67027ed3
Sep 28 02:06:37 dist-test-slave-23bp krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.182.131@KRBTEST.COM; defaulting to no policy
I20260928 02:06:37.840729 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:37.807646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:33071 (local address 127.1.182.190:39053)
0928 02:06:37.822903 (+ 15257us) server_negotiation.cc:206] Beginning negotiation
0928 02:06:37.822909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:06:37.822934 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:06:37.824026 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:06:37.824033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:06:37.824119 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:06:37.824253 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:06:37.825835 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:37.826583 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:37.828246 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:37.828643 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:37.828780 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:37.832133 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:06:37.832138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:06:37.832142 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:06:37.832173 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:06:37.835049 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:06:37.838271 (+ 3222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:06:37.838284 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:06:37.838287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:06:37.838410 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:06:37.838857 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:06:37.838869 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:06:37.838871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:06:37.839213 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:06:37.839374 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 02:06:37.840073 (+ 699us) server_negotiation.cc:299] Negotiation successful
0928 02:06:37.840298 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15121,"thread_start_us":15285,"threads_started":1}
I20260928 02:06:37.841899 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:37.808382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39053 (local address 127.1.182.130:33071)
0928 02:06:37.820449 (+ 12067us) negotiation.cc:107] Waiting for socket to connect
0928 02:06:37.820523 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 02:06:37.823631 (+ 3108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:06:37.824547 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:06:37.824582 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:06:37.825047 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:06:37.825631 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:06:37.825647 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:06:37.826755 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:06:37.826761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:06:37.828094 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:06:37.828106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:06:37.828880 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:06:37.828886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:06:37.829109 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:37.829814 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:06:37.829835 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:06:37.831912 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:06:37.836921 (+ 5009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:06:37.836979 (+ 58us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:06:37.836997 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:06:37.838136 (+ 1139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:06:37.838532 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:06:37.838537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:06:37.838541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:06:37.838737 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:06:37.839340 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:06:37.839348 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:06:37.839702 (+ 354us) client_negotiation.cc:769] Sending connection context
0928 02:06:37.839968 (+ 266us) client_negotiation.cc:240] Negotiation successful
0928 02:06:37.840505 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11633,"thread_start_us":11379,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.131@KRBTEST.COM" created.
I20260928 02:06:37.843472 13446 heartbeater.cc:347] Connected to a master server at 127.1.182.190:39053
I20260928 02:06:37.843818 13446 heartbeater.cc:464] Registering TS with master...
I20260928 02:06:37.844676 13446 heartbeater.cc:511] Master 127.1.182.190:39053 requested a full tablet report, sending...
I20260928 02:06:37.846400 13087 ts_manager.cc:194] Registered new tserver with Master: 92adcc0a73274c0aa8d163213996f4a6 (127.1.182.130:33109)
I20260928 02:06:37.847642 13087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:33071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
I20260928 02:06:37.876652 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.182.148:59189/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39053
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:06:37.989977 13301 heartbeater.cc:503] Master 127.1.182.190:39053 was elected leader, sending a full tablet report...
W20260928 02:06:38.237437 13452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:06:38.237747 13452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:06:38.237917 13452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:06:38.245677 13452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:06:38.245836 13452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:06:38.246057 13452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:06:38.255755 13452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.182.148:59189/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:06:38.257054 13452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:06:38.258409 13452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:06:38.271847 13457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:06:38.271819 13460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:06:38.271903 13458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:06:38.272878 13452 server_base.cc:1034] running on GCE node
I20260928 02:06:38.273449 13452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:06:38.274358 13452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:06:38.275576 13452 hybrid_clock.cc:648] HybridClock initialized: now 1790561198275534 us; error 54 us; skew 500 ppm
I20260928 02:06:38.275899 13452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:06:38 dist-test-slave-23bp krb5kdc[12607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561198, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:06:38.284410 13452 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:06:38.285928 13452 webserver.cc:460] Webserver started at http://127.1.182.131:44601/ using document root <none> and password file <none>
I20260928 02:06:38.286603 13452 fs_manager.cc:359] Metadata directory not provided
I20260928 02:06:38.286746 13452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:06:38.287557 13452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:06:38 dist-test-slave-23bp krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561198, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
I20260928 02:06:38.302084 13452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "295768432ed2447db66e8936f398a8b6"
format_stamp: "Formatted at 2026-09-28 02:06:38 on dist-test-slave-23bp"
server_key: "a37afc2576f123e8e84d63b0efdd43bb"
server_key_iv: "343f8ae1721840c1b1245b38dca7831e"
server_key_version: "kuduclusterkey@0"
I20260928 02:06:38.303093 13452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "295768432ed2447db66e8936f398a8b6"
format_stamp: "Formatted at 2026-09-28 02:06:38 on dist-test-slave-23bp"
server_key: "a37afc2576f123e8e84d63b0efdd43bb"
server_key_iv: "343f8ae1721840c1b1245b38dca7831e"
server_key_version: "kuduclusterkey@0"
I20260928 02:06:38.309180 13452 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 02:06:38.325975 13467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:06:38.327739 13452 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.005s
I20260928 02:06:38.327940 13452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "295768432ed2447db66e8936f398a8b6"
format_stamp: "Formatted at 2026-09-28 02:06:38 on dist-test-slave-23bp"
server_key: "a37afc2576f123e8e84d63b0efdd43bb"
server_key_iv: "343f8ae1721840c1b1245b38dca7831e"
server_key_version: "kuduclusterkey@0"
I20260928 02:06:38.328176 13452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:06:38.343992 13452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:06:38.347796 13452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:06:38.348099 13452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:06:38.349421 13452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:06:38.350791 13452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:06:38.350891 13452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:06:38.351032 13452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:06:38.351117 13452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 02:06:38.385351 13452 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:42165
I20260928 02:06:38.385385 13579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:42165 every 8 connection(s)
I20260928 02:06:38.386674 13452 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:06:38.393532 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 13452
I20260928 02:06:38.393749 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:06:38.394243 1754 external_mini_cluster.cc:1428] Setting key 8950d60f5cdb09c2c267499ac5f76991
Sep 28 02:06:38 dist-test-slave-23bp krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561198, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:06:38.407505 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:38.389205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:36917 (local address 127.1.182.190:39053)
0928 02:06:38.390120 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0928 02:06:38.390128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:06:38.391149 (+ 1021us) server_negotiation.cc:407] Connection header received
0928 02:06:38.392915 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:06:38.392920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:06:38.392994 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:06:38.393090 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:06:38.394695 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.395704 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.397312 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.397749 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.397879 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:38.401522 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:06:38.401529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:06:38.401534 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:06:38.401573 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:06:38.404472 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:06:38.405207 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:06:38.405215 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:06:38.405219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:06:38.405345 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:06:38.405775 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:06:38.405782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:06:38.405785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:06:38.406055 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:06:38.406210 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:06:38.406895 (+ 685us) server_negotiation.cc:299] Negotiation successful
0928 02:06:38.407131 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":652,"threads_started":1}
I20260928 02:06:38.408669 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:38.390182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39053 (local address 127.1.182.131:36917)
0928 02:06:38.390965 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0928 02:06:38.391009 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:06:38.392662 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:06:38.393378 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:06:38.393398 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:06:38.393918 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:06:38.394529 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:06:38.394547 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.395862 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.395872 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:06:38.397164 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:06:38.397175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.397959 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.397968 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:06:38.398212 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:38.398904 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:06:38.398925 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:06:38.401298 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:06:38.404626 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:06:38.404635 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:06:38.404650 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:06:38.405053 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:06:38.405452 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:06:38.405458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:06:38.405462 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:06:38.405663 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:06:38.406191 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:06:38.406199 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:06:38.406541 (+ 342us) client_negotiation.cc:769] Sending connection context
0928 02:06:38.406825 (+ 284us) client_negotiation.cc:240] Negotiation successful
0928 02:06:38.407348 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":366,"threads_started":1}
I20260928 02:06:38.410203 13581 heartbeater.cc:347] Connected to a master server at 127.1.182.190:39053
I20260928 02:06:38.410471 13581 heartbeater.cc:464] Registering TS with master...
I20260928 02:06:38.411402 13581 heartbeater.cc:511] Master 127.1.182.190:39053 requested a full tablet report, sending...
I20260928 02:06:38.412773 13087 ts_manager.cc:194] Registered new tserver with Master: 295768432ed2447db66e8936f398a8b6 (127.1.182.131:42165)
I20260928 02:06:38.413513 13087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:36917
I20260928 02:06:38.421900 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:06:38.439527 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:38.426737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46684 (local address 127.1.182.190:39053)
0928 02:06:38.426922 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:06:38.426930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:06:38.427381 (+ 451us) server_negotiation.cc:407] Connection header received
0928 02:06:38.427609 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:06:38.427614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:06:38.427697 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:06:38.427805 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:06:38.429229 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.429941 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.431369 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.431845 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.431988 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:38.434267 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:06:38.434271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:06:38.434275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:06:38.434304 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:06:38.436946 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:06:38.437652 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:06:38.437656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:06:38.437658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:06:38.437773 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:06:38.438085 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:06:38.438089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:06:38.438092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:06:38.438338 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:06:38.438586 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 02:06:38.438962 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 02:06:38.439213 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:06:38.443432 13087 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:06:38.450839 13087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:06:38.474449 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:38.463849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:33109 (local address 127.0.0.1:41406)
0928 02:06:38.464657 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0928 02:06:38.464694 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:06:38.464840 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:06:38.465936 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:06:38.465948 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:06:38.465988 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:06:38.466842 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:06:38.466867 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.468843 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.468848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:06:38.471360 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:06:38.471379 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.473474 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.473480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:06:38.473672 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:38.473720 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 02:06:38.473847 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 02:06:38.474042 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":348,"threads_started":1}
I20260928 02:06:38.475389 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:38.463955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41406 (local address 127.1.182.130:33109)
0928 02:06:38.464660 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0928 02:06:38.464669 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:06:38.464795 (+ 126us) server_negotiation.cc:407] Connection header received
0928 02:06:38.465049 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:06:38.465058 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:06:38.465367 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:06:38.465613 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:06:38.467101 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.468696 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.471551 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.472988 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.473112 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:38.473328 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 02:06:38.474859 (+ 1531us) server_negotiation.cc:299] Negotiation successful
0928 02:06:38.475034 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":444,"threads_started":1}
I20260928 02:06:38.475646 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:38.465055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:44035 (local address 127.0.0.1:57128)
0928 02:06:38.467150 (+ 2095us) negotiation.cc:107] Waiting for socket to connect
0928 02:06:38.467176 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:06:38.467295 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:06:38.468218 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:06:38.468222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:06:38.468245 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:06:38.468630 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:06:38.468643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.470581 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.470586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:06:38.472735 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:06:38.472749 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.474856 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.474862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:06:38.475033 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:38.475100 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 02:06:38.475186 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:06:38.475302 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1961,"thread_start_us":1810,"threads_started":1}
I20260928 02:06:38.476289 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:38.465305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.1.182.129:44035)
0928 02:06:38.466824 (+ 1519us) server_negotiation.cc:206] Beginning negotiation
0928 02:06:38.466833 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:06:38.467433 (+ 600us) server_negotiation.cc:407] Connection header received
0928 02:06:38.467543 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:06:38.467551 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:06:38.467794 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:06:38.467968 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:06:38.468836 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.470421 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.472917 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.474330 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.474465 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:38.474708 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 02:06:38.475774 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0928 02:06:38.475935 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1344,"thread_start_us":1204,"threads_started":1}
I20260928 02:06:38.480731 13236 tablet_service.cc:1467] Processing CreateTablet for tablet d39135a6b0d34242a7dc6974bc558461 (DEFAULT_TABLE table=test-table [id=3caa21f0f097482bbeda95101bff647f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:06:38.480970 13380 tablet_service.cc:1467] Processing CreateTablet for tablet d39135a6b0d34242a7dc6974bc558461 (DEFAULT_TABLE table=test-table [id=3caa21f0f097482bbeda95101bff647f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:06:38.482283 13236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d39135a6b0d34242a7dc6974bc558461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:06:38.482363 13380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d39135a6b0d34242a7dc6974bc558461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:06:38.484331 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:38.467243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:42165 (local address 127.0.0.1:34734)
0928 02:06:38.475974 (+ 8731us) negotiation.cc:107] Waiting for socket to connect
0928 02:06:38.475993 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:06:38.476096 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:06:38.477578 (+ 1482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:06:38.477582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:06:38.477606 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:06:38.478198 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:06:38.478209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.480110 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.480115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:06:38.481864 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:06:38.481881 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.483629 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.483635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:06:38.483818 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:38.483853 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:06:38.483948 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 02:06:38.484077 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8644,"thread_start_us":9450,"threads_started":1}
I20260928 02:06:38.484866 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:38.467584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34734 (local address 127.1.182.131:42165)
0928 02:06:38.476982 (+ 9398us) server_negotiation.cc:206] Beginning negotiation
0928 02:06:38.476989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:06:38.477016 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:06:38.477095 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:06:38.477101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:06:38.477365 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:06:38.477574 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:06:38.478400 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.479952 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.482045 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.483457 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.483602 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:38.483871 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 02:06:38.483984 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 02:06:38.484148 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9249,"thread_start_us":9064,"threads_started":1}
I20260928 02:06:38.488741 13515 tablet_service.cc:1467] Processing CreateTablet for tablet d39135a6b0d34242a7dc6974bc558461 (DEFAULT_TABLE table=test-table [id=3caa21f0f097482bbeda95101bff647f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:06:38.490188 13515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d39135a6b0d34242a7dc6974bc558461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:06:38.497117 13600 tablet_bootstrap.cc:492] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572: Bootstrap starting.
I20260928 02:06:38.502800 13600 tablet_bootstrap.cc:654] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572: Neither blocks nor log segments found. Creating new log.
I20260928 02:06:38.504716 13600 log.cc:824] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572: Log is configured to *not* fsync() on all Append() calls
I20260928 02:06:38.509141 13602 tablet_bootstrap.cc:492] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6: Bootstrap starting.
I20260928 02:06:38.515935 13602 tablet_bootstrap.cc:654] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6: Neither blocks nor log segments found. Creating new log.
I20260928 02:06:38.516840 13601 tablet_bootstrap.cc:492] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6: Bootstrap starting.
I20260928 02:06:38.520998 13601 tablet_bootstrap.cc:654] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6: Neither blocks nor log segments found. Creating new log.
I20260928 02:06:38.521170 13602 log.cc:824] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:06:38.521957 13600 tablet_bootstrap.cc:492] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572: No bootstrap required, opened a new log
I20260928 02:06:38.522333 13600 ts_tablet_manager.cc:1397] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.011s
I20260928 02:06:38.522317 13601 log.cc:824] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:06:38.528481 13601 tablet_bootstrap.cc:492] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6: No bootstrap required, opened a new log
I20260928 02:06:38.528826 13601 ts_tablet_manager.cc:1397] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260928 02:06:38.534689 13602 tablet_bootstrap.cc:492] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6: No bootstrap required, opened a new log
I20260928 02:06:38.535112 13602 ts_tablet_manager.cc:1397] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.008s
I20260928 02:06:38.536224 13600 raft_consensus.cc:348] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92adcc0a73274c0aa8d163213996f4a6" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33109 } } peers { permanent_uuid: "fa9acde1b66e4e7cb200c97145d4e572" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44035 } } peers { permanent_uuid: "295768432ed2447db66e8936f398a8b6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 42165 } }
I20260928 02:06:38.536814 13600 raft_consensus.cc:374] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:06:38.536955 13600 raft_consensus.cc:729] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9acde1b66e4e7cb200c97145d4e572, State: Initialized, Role: FOLLOWER
I20260928 02:06:38.537819 13600 consensus_queue.cc:260] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92adcc0a73274c0aa8d163213996f4a6" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33109 } } peers { permanent_uuid: "fa9acde1b66e4e7cb200c97145d4e572" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44035 } } peers { permanent_uuid: "295768432ed2447db66e8936f398a8b6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 42165 } }
I20260928 02:06:38.540501 13601 raft_consensus.cc:348] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92adcc0a73274c0aa8d163213996f4a6" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33109 } } peers { permanent_uuid: "fa9acde1b66e4e7cb200c97145d4e572" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44035 } } peers { permanent_uuid: "295768432ed2447db66e8936f398a8b6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 42165 } }
I20260928 02:06:38.540994 13601 raft_consensus.cc:374] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:06:38.541157 13601 raft_consensus.cc:729] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 295768432ed2447db66e8936f398a8b6, State: Initialized, Role: FOLLOWER
I20260928 02:06:38.541819 13601 consensus_queue.cc:260] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92adcc0a73274c0aa8d163213996f4a6" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33109 } } peers { permanent_uuid: "fa9acde1b66e4e7cb200c97145d4e572" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44035 } } peers { permanent_uuid: "295768432ed2447db66e8936f398a8b6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 42165 } }
I20260928 02:06:38.542969 13581 heartbeater.cc:503] Master 127.1.182.190:39053 was elected leader, sending a full tablet report...
I20260928 02:06:38.543804 13601 ts_tablet_manager.cc:1428] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260928 02:06:38.545284 13600 ts_tablet_manager.cc:1428] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572: Time spent starting tablet: real 0.023s user 0.005s sys 0.009s
I20260928 02:06:38.547631 13602 raft_consensus.cc:348] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92adcc0a73274c0aa8d163213996f4a6" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33109 } } peers { permanent_uuid: "fa9acde1b66e4e7cb200c97145d4e572" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44035 } } peers { permanent_uuid: "295768432ed2447db66e8936f398a8b6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 42165 } }
I20260928 02:06:38.548105 13602 raft_consensus.cc:374] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:06:38.548200 13602 raft_consensus.cc:729] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92adcc0a73274c0aa8d163213996f4a6, State: Initialized, Role: FOLLOWER
I20260928 02:06:38.548846 13602 consensus_queue.cc:260] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92adcc0a73274c0aa8d163213996f4a6" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33109 } } peers { permanent_uuid: "fa9acde1b66e4e7cb200c97145d4e572" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44035 } } peers { permanent_uuid: "295768432ed2447db66e8936f398a8b6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 42165 } }
I20260928 02:06:38.549945 13446 heartbeater.cc:503] Master 127.1.182.190:39053 was elected leader, sending a full tablet report...
I20260928 02:06:38.550742 13602 ts_tablet_manager.cc:1428] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
W20260928 02:06:38.554982 13447 tablet.cc:2369] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:06:38.638281 13582 tablet.cc:2369] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:06:38.703470 13302 tablet.cc:2369] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:06:38.899320 13608 raft_consensus.cc:482] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:06:38.899617 13608 raft_consensus.cc:504] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92adcc0a73274c0aa8d163213996f4a6" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33109 } } peers { permanent_uuid: "fa9acde1b66e4e7cb200c97145d4e572" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44035 } } peers { permanent_uuid: "295768432ed2447db66e8936f398a8b6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 42165 } }
I20260928 02:06:38.901319 13608 leader_election.cc:288] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92adcc0a73274c0aa8d163213996f4a6 (127.1.182.130:33109), fa9acde1b66e4e7cb200c97145d4e572 (127.1.182.129:44035)
I20260928 02:06:38.908110 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:38.901590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:33109 (local address 127.1.182.131:51189)
0928 02:06:38.901794 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 02:06:38.901814 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:06:38.901957 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:06:38.902434 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:06:38.902439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:06:38.902474 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:06:38.902664 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:06:38.902674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.903910 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.903916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:06:38.905896 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:06:38.905910 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.907370 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.907376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:06:38.907557 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:38.907581 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:06:38.907693 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 02:06:38.907827 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 02:06:38.908102 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:38.901796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:51189 (local address 127.1.182.130:33109)
0928 02:06:38.902002 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:06:38.902010 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:06:38.902046 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:06:38.902122 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:06:38.902126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:06:38.902207 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:06:38.902318 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:06:38.902826 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.903765 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.906090 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.907215 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.907381 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:38.907454 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:06:38.907726 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 02:06:38.907852 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:06:38.909152 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:38.902660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:40253 (local address 127.1.182.129:44035)
0928 02:06:38.902855 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 02:06:38.902861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:06:38.902879 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:06:38.902942 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:06:38.902947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:06:38.903011 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:06:38.903157 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:06:38.903757 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.904912 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.906885 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.907775 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.907959 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:38.908030 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:06:38.908352 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 02:06:38.908521 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:06:38.909726 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:38.901939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:44035 (local address 127.1.182.131:40253)
0928 02:06:38.902527 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0928 02:06:38.902548 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:06:38.902657 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:06:38.903199 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:06:38.903204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:06:38.903227 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:06:38.903539 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:06:38.903548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.905095 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.905101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:06:38.906740 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:06:38.906752 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.907974 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.907980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:06:38.908140 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:38.908162 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:06:38.909333 (+ 1171us) client_negotiation.cc:240] Negotiation successful
0928 02:06:38.909449 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":303,"threads_started":1}
I20260928 02:06:38.909730 13400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39135a6b0d34242a7dc6974bc558461" candidate_uuid: "295768432ed2447db66e8936f398a8b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92adcc0a73274c0aa8d163213996f4a6" is_pre_election: true
I20260928 02:06:38.910394 13400 raft_consensus.cc:2454] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 295768432ed2447db66e8936f398a8b6 in term 0.
I20260928 02:06:38.910796 13256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39135a6b0d34242a7dc6974bc558461" candidate_uuid: "295768432ed2447db66e8936f398a8b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9acde1b66e4e7cb200c97145d4e572" is_pre_election: true
I20260928 02:06:38.911051 13607 raft_consensus.cc:482] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:06:38.911144 13471 leader_election.cc:302] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 295768432ed2447db66e8936f398a8b6, 92adcc0a73274c0aa8d163213996f4a6; no voters:
I20260928 02:06:38.911345 13607 raft_consensus.cc:504] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92adcc0a73274c0aa8d163213996f4a6" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33109 } } peers { permanent_uuid: "fa9acde1b66e4e7cb200c97145d4e572" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44035 } } peers { permanent_uuid: "295768432ed2447db66e8936f398a8b6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 42165 } }
I20260928 02:06:38.911599 13608 raft_consensus.cc:2781] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:06:38.911765 13608 raft_consensus.cc:482] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:06:38.911903 13608 raft_consensus.cc:3037] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:06:38.912065 13256 raft_consensus.cc:2454] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 295768432ed2447db66e8936f398a8b6 in term 0.
I20260928 02:06:38.913154 13607 leader_election.cc:288] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92adcc0a73274c0aa8d163213996f4a6 (127.1.182.130:33109), 295768432ed2447db66e8936f398a8b6 (127.1.182.131:42165)
I20260928 02:06:38.915155 13608 raft_consensus.cc:504] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92adcc0a73274c0aa8d163213996f4a6" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33109 } } peers { permanent_uuid: "fa9acde1b66e4e7cb200c97145d4e572" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44035 } } peers { permanent_uuid: "295768432ed2447db66e8936f398a8b6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 42165 } }
I20260928 02:06:38.916162 13400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39135a6b0d34242a7dc6974bc558461" candidate_uuid: "295768432ed2447db66e8936f398a8b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92adcc0a73274c0aa8d163213996f4a6"
I20260928 02:06:38.916400 13256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39135a6b0d34242a7dc6974bc558461" candidate_uuid: "295768432ed2447db66e8936f398a8b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9acde1b66e4e7cb200c97145d4e572"
I20260928 02:06:38.916625 13256 raft_consensus.cc:3037] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:06:38.916822 13400 raft_consensus.cc:3037] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:06:38.919167 13400 raft_consensus.cc:2454] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 295768432ed2447db66e8936f398a8b6 in term 1.
I20260928 02:06:38.919636 13471 leader_election.cc:302] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 295768432ed2447db66e8936f398a8b6, 92adcc0a73274c0aa8d163213996f4a6; no voters:
I20260928 02:06:38.919900 13608 leader_election.cc:288] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 [CANDIDATE]: Term 1 election: Requested vote from peers 92adcc0a73274c0aa8d163213996f4a6 (127.1.182.130:33109), fa9acde1b66e4e7cb200c97145d4e572 (127.1.182.129:44035)
I20260928 02:06:38.920127 13608 raft_consensus.cc:2781] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:06:38.920291 13608 raft_consensus.cc:686] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 [term 1 LEADER]: Becoming Leader. State: Replica: 295768432ed2447db66e8936f398a8b6, State: Running, Role: LEADER
I20260928 02:06:38.920805 13256 raft_consensus.cc:2454] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 295768432ed2447db66e8936f398a8b6 in term 1.
I20260928 02:06:38.921631 13608 consensus_queue.cc:237] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92adcc0a73274c0aa8d163213996f4a6" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33109 } } peers { permanent_uuid: "fa9acde1b66e4e7cb200c97145d4e572" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44035 } } peers { permanent_uuid: "295768432ed2447db66e8936f398a8b6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 42165 } }
I20260928 02:06:38.922482 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:38.913613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:50891 (local address 127.1.182.130:33109)
0928 02:06:38.913876 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 02:06:38.913884 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:06:38.914035 (+ 151us) server_negotiation.cc:407] Connection header received
0928 02:06:38.914307 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:06:38.914313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:06:38.914380 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:06:38.914436 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:06:38.915233 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.916251 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.918922 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.920688 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.920860 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:38.921012 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:06:38.922099 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0928 02:06:38.922229 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260928 02:06:38.922879 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:38.913335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:33109 (local address 127.1.182.129:50891)
0928 02:06:38.913928 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0928 02:06:38.913951 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:06:38.914141 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:06:38.914625 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:06:38.914630 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:06:38.914653 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:06:38.915011 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:06:38.915023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.916453 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.916457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:06:38.918750 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:06:38.918764 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.921197 (+ 2433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.921210 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:06:38.921398 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:38.921433 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:06:38.921525 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 02:06:38.921655 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":275,"threads_started":1}
I20260928 02:06:38.923852 13400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39135a6b0d34242a7dc6974bc558461" candidate_uuid: "fa9acde1b66e4e7cb200c97145d4e572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92adcc0a73274c0aa8d163213996f4a6" is_pre_election: true
I20260928 02:06:38.924158 13400 raft_consensus.cc:2379] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa9acde1b66e4e7cb200c97145d4e572 in current term 1: Already voted for candidate 295768432ed2447db66e8936f398a8b6 in this term.
I20260928 02:06:38.924981 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:38.914068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:56651 (local address 127.1.182.131:42165)
0928 02:06:38.914884 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0928 02:06:38.914888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:06:38.914900 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:06:38.914939 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:06:38.914943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:06:38.914993 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:06:38.915133 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:06:38.915713 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.917147 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.922504 (+ 5357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.923877 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.924026 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:38.924093 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:06:38.924600 (+ 507us) server_negotiation.cc:299] Negotiation successful
0928 02:06:38.924735 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709}
I20260928 02:06:38.925622 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:38.913995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:42165 (local address 127.1.182.129:56651)
0928 02:06:38.914584 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 02:06:38.914598 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:06:38.914686 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:06:38.915146 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:06:38.915150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:06:38.915171 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:06:38.915552 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:06:38.915561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.919440 (+ 3879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.919445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:06:38.922323 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:06:38.922337 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.924251 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.924257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:06:38.924418 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:38.924448 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 02:06:38.925119 (+ 671us) client_negotiation.cc:240] Negotiation successful
0928 02:06:38.925267 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":300,"threads_started":1}
I20260928 02:06:38.927613 13535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39135a6b0d34242a7dc6974bc558461" candidate_uuid: "fa9acde1b66e4e7cb200c97145d4e572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "295768432ed2447db66e8936f398a8b6" is_pre_election: true
I20260928 02:06:38.927971 13086 catalog_manager.cc:5495] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 295768432ed2447db66e8936f398a8b6 (127.1.182.131). New cstate: current_term: 1 leader_uuid: "295768432ed2447db66e8936f398a8b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92adcc0a73274c0aa8d163213996f4a6" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa9acde1b66e4e7cb200c97145d4e572" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "295768432ed2447db66e8936f398a8b6" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 42165 } health_report { overall_health: HEALTHY } } }
I20260928 02:06:38.928668 13192 leader_election.cc:302] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa9acde1b66e4e7cb200c97145d4e572; no voters: 295768432ed2447db66e8936f398a8b6, 92adcc0a73274c0aa8d163213996f4a6
I20260928 02:06:38.929072 13607 raft_consensus.cc:2726] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:06:38.945746 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:38.939931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34746 (local address 127.1.182.131:42165)
0928 02:06:38.940107 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 02:06:38.940115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:06:38.940604 (+ 489us) server_negotiation.cc:407] Connection header received
0928 02:06:38.940854 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:06:38.940858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:06:38.940930 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:06:38.941077 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:06:38.941457 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.942177 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.943675 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:38.944060 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:38.944202 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:38.944688 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:06:38.945162 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:06:38.945329 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:06:38.945393 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 02:06:38.945468 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:06:38.953926 13400 raft_consensus.cc:1261] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6 [term 1 FOLLOWER]: Refusing update from remote peer 295768432ed2447db66e8936f398a8b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:06:38.953971 13256 raft_consensus.cc:1261] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572 [term 1 FOLLOWER]: Refusing update from remote peer 295768432ed2447db66e8936f398a8b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:06:38.954821 13608 consensus_queue.cc:1035] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92adcc0a73274c0aa8d163213996f4a6" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 33109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:06:38.955133 13613 consensus_queue.cc:1035] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa9acde1b66e4e7cb200c97145d4e572" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:06:38.983839 13086 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:06:38.984171 13086 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:06:38.987552 13086 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 038426ad89b54a0cac58d233a14e07e7: Sending DeleteTablet for 3 replicas of tablet d39135a6b0d34242a7dc6974bc558461
I20260928 02:06:38.988663 13380 tablet_service.cc:1514] Processing DeleteTablet for tablet d39135a6b0d34242a7dc6974bc558461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:06:38 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:41406
I20260928 02:06:38.988796 13515 tablet_service.cc:1514] Processing DeleteTablet for tablet d39135a6b0d34242a7dc6974bc558461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:06:38 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:34734
I20260928 02:06:38.988855 13236 tablet_service.cc:1514] Processing DeleteTablet for tablet d39135a6b0d34242a7dc6974bc558461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:06:38 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:57128
I20260928 02:06:38.990986 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 13167
I20260928 02:06:39.006696 13629 tablet_replica.cc:330] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572: stopping tablet replica
I20260928 02:06:39.007510 13629 raft_consensus.cc:2229] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:06:39.008231 13629 raft_consensus.cc:2258] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:06:39.008867 13627 tablet_replica.cc:330] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6: stopping tablet replica
I20260928 02:06:39.009377 13627 raft_consensus.cc:2229] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:06:39.009881 13627 raft_consensus.cc:2258] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:06:39.010085 13628 tablet_replica.cc:330] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6: stopping tablet replica
I20260928 02:06:39.010552 13628 raft_consensus.cc:2229] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:06:39.011735 13628 raft_consensus.cc:2258] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:06:39.012982 13629 ts_tablet_manager.cc:1905] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:06:39.014377 13628 ts_tablet_manager.cc:1905] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:06:39.014747 13627 ts_tablet_manager.cc:1905] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 02:06:39 dist-test-slave-23bp krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:06:39.020840 13629 ts_tablet_manager.cc:1918] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:06:39.021027 13629 log.cc:1196] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-0/wal/wals/d39135a6b0d34242a7dc6974bc558461
I20260928 02:06:39.021596 13629 ts_tablet_manager.cc:1939] T d39135a6b0d34242a7dc6974bc558461 P fa9acde1b66e4e7cb200c97145d4e572: Deleting consensus metadata
I20260928 02:06:39.022519 13628 ts_tablet_manager.cc:1918] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:06:39.022691 13628 log.cc:1196] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/d39135a6b0d34242a7dc6974bc558461
I20260928 02:06:39.023272 13628 ts_tablet_manager.cc:1939] T d39135a6b0d34242a7dc6974bc558461 P 295768432ed2447db66e8936f398a8b6: Deleting consensus metadata
I20260928 02:06:39.023634 13074 catalog_manager.cc:4841] TS fa9acde1b66e4e7cb200c97145d4e572 (127.1.182.129:44035): tablet d39135a6b0d34242a7dc6974bc558461 (table test-table [id=3caa21f0f097482bbeda95101bff647f]) successfully deleted
I20260928 02:06:39.024927 13074 catalog_manager.cc:4841] TS 295768432ed2447db66e8936f398a8b6 (127.1.182.131:42165): tablet d39135a6b0d34242a7dc6974bc558461 (table test-table [id=3caa21f0f097482bbeda95101bff647f]) successfully deleted
I20260928 02:06:39.026292 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:38.993744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57144 (local address 127.1.182.129:44035)
0928 02:06:38.993976 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 02:06:38.993986 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:06:38.998035 (+ 4049us) server_negotiation.cc:407] Connection header received
0928 02:06:38.999100 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:06:38.999111 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:06:38.999225 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:06:38.999320 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:06:39.004282 (+ 4962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:39.005826 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:39.009663 (+ 3837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:39.010342 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:39.010502 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:39.017081 (+ 6579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:06:39.017096 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:06:39.017113 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:06:39.017164 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:06:39.021981 (+ 4817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:06:39.023169 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:06:39.023179 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:06:39.023183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:06:39.023429 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:06:39.023999 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:06:39.024004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:06:39.024007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:06:39.024473 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:06:39.025188 (+ 715us) server_negotiation.cc:1035] Waiting for connection context
0928 02:06:39.025691 (+ 503us) server_negotiation.cc:299] Negotiation successful
0928 02:06:39.025905 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:06:39.026921 13296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:06:39.035197 13627 ts_tablet_manager.cc:1918] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:06:39.035396 13627 log.cc:1196] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/d39135a6b0d34242a7dc6974bc558461
I20260928 02:06:39.035948 13627 ts_tablet_manager.cc:1939] T d39135a6b0d34242a7dc6974bc558461 P 92adcc0a73274c0aa8d163213996f4a6: Deleting consensus metadata
I20260928 02:06:39.037911 13074 catalog_manager.cc:4841] TS 92adcc0a73274c0aa8d163213996f4a6 (127.1.182.130:33109): tablet d39135a6b0d34242a7dc6974bc558461 (table test-table [id=3caa21f0f097482bbeda95101bff647f]) successfully deleted
I20260928 02:06:39.119319 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 13167
I20260928 02:06:39.141996 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 13307
Sep 28 02:06:39 dist-test-slave-23bp krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.130@KRBTEST.COM
I20260928 02:06:39.158664 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:39.142841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.1.182.130:33109)
0928 02:06:39.143022 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:06:39.143029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:06:39.143050 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:06:39.143298 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:06:39.143305 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:06:39.143397 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:06:39.143508 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:06:39.144742 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:39.145883 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:39.147210 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:39.147856 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:39.147986 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:39.152286 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:06:39.152305 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:06:39.152326 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:06:39.152368 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:06:39.155991 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:06:39.156657 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:06:39.156666 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:06:39.156670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:06:39.156823 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:06:39.157149 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:06:39.157154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:06:39.157158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:06:39.157588 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:06:39.157876 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 02:06:39.158105 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 02:06:39.158307 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:06:39.159391 13440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:06:39.249856 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 13307
I20260928 02:06:39.272579 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 13452
Sep 28 02:06:39 dist-test-slave-23bp krb5kdc[12607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.131@KRBTEST.COM
I20260928 02:06:39.289808 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:06:39.273358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34754 (local address 127.1.182.131:42165)
0928 02:06:39.273594 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 02:06:39.273601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:06:39.273687 (+ 86us) server_negotiation.cc:407] Connection header received
0928 02:06:39.273914 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:06:39.273923 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:06:39.274006 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:06:39.274126 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:06:39.275938 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:39.276680 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:39.277937 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:06:39.278406 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:06:39.278547 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:06:39.283949 (+ 5402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:06:39.283968 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:06:39.283986 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:06:39.284018 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:06:39.287008 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:06:39.287744 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:06:39.287759 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:06:39.287762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:06:39.287885 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:06:39.288275 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:06:39.288280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:06:39.288289 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:06:39.288634 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:06:39.288960 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0928 02:06:39.289259 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 02:06:39.289478 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:06:39.290513 13575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:06:39.418180 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 13452
I20260928 02:06:39.436905 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 13055
I20260928 02:06:39.438063 13117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:06:39.549775 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 13055
I20260928 02:06:39.576239 1754 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 02:06:39.911398 1754 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 02:06:39.911644 1754 mini_ranger.cc:66] Stopping Ranger...
I20260928 02:06:40.257534 1754 mini_ranger.cc:68] Stopped Ranger
2026-09-28 02:06:40.257 UTC [12638] LOG: received smart shutdown request
2026-09-28 02:06:40.262 UTC [12638] LOG: background worker "logical replication launcher" (PID 12647) exited with exit code 1
2026-09-28 02:06:40.262 UTC [12642] LOG: shutting down
2026-09-28 02:06:40.427 UTC [12638] LOG: database system is shut down
2026-09-28T02:06:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48632 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:06:40 dist-test-slave-23bp krb5kdc[13643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:06:40 dist-test-slave-23bp krb5kdc[13643](info): set up 2 sockets
Sep 28 02:06:40 dist-test-slave-23bp krb5kdc[13643](info): commencing operation
krb5kdc: starting...
W20260928 02:06:41.666574 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:06:41 dist-test-slave-23bp krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:06:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:06:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.182.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.182.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.182.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/rangeradmin_127.1.182.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.182.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/rangeradmin_127.1.182.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.182.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.182.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.182.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/rangerlookup_127.1.182.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.182.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/rangerlookup_127.1.182.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.182.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.182.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.182.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/HTTP_127.1.182.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.182.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/HTTP_127.1.182.148@KRBTEST.COM.keytab.
I20260928 02:06:41.998136 1754 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasktsFKfV/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/postgres -l logfile start
2026-09-28 02:06:44.605 UTC [13673] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 02:06:44.605 UTC [13673] LOG: listening on IPv4 address "127.1.182.148", port 34387
2026-09-28 02:06:44.608 UTC [13673] LOG: listening on Unix socket "/tmp/.s.PGSQL.34387"
2026-09-28 02:06:44.630 UTC [13676] LOG: database system was shut down at 2026-09-28 02:06:43 UTC
2026-09-28 02:06:44.633 UTC [13673] LOG: database system is ready to accept connections
I20260928 02:06:45.597675 1754 mini_postgres.cc:96] Postgres bound to 34387
2026-09-28 02:06:45.620 UTC [13684] FATAL: database "slave" does not exist
127.1.182.148:34387 - accepting connections
I20260928 02:06:45.620855 1754 mini_ranger.cc:161] Starting Ranger...
I20260928 02:06:45.647967 1754 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 02:06:46.155227 1754 mini_ranger.cc:87] Created ranger Postgres database
I20260928 02:06:46.155360 1754 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ranger-admin
2026-09-28 02:06:46,458 [I] DB FLAVOR :POSTGRES
2026-09-28 02:06:46,458 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:06:46,458 [I] Checking connection..
2026-09-28 02:06:46,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 02:06:46,701 [I] Checking connection passed.
2026-09-28 02:06:46,701 [I] --------- Verifying version history table ---------
2026-09-28 02:06:46,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:06:46,945 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:06:46,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:06:47,173 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:06:47,174 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 02:06:47,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 02:06:47.426 UTC [13748] WARNING: there is no transaction in progress
2026-09-28 02:06:47,432 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 02:06:47,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:06:47,679 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 02:06:47,679 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:06:47,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 02:06:47,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 02:06:48,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-23bp.c.gcp-upstream.internal','N') ;"
2026-09-28 02:06:48,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:06:48,643 [I] Table x_portal_user does not exist in database ranger
2026-09-28 02:06:48,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:06:48,885 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 02:06:48,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:06:49,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:06:49,401 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 02:06:49,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 02:06:50.395 UTC [13856] WARNING: there is no transaction in progress
2026-09-28 02:06:50.411 UTC [13856] WARNING: there is no transaction in progress
2026-09-28 02:06:50.427 UTC [13856] WARNING: there is no transaction in progress
2026-09-28 02:06:50.441 UTC [13856] WARNING: there is no transaction in progress
2026-09-28 02:06:50.460 UTC [13856] WARNING: there is no transaction in progress
2026-09-28 02:06:50.478 UTC [13856] WARNING: there is no transaction in progress
2026-09-28 02:06:50.538 UTC [13856] WARNING: there is no transaction in progress
2026-09-28 02:06:50.545 UTC [13856] WARNING: there is no transaction in progress
2026-09-28 02:06:50.560 UTC [13856] WARNING: there is no transaction in progress
2026-09-28 02:06:50.571 UTC [13856] WARNING: there is no transaction in progress
2026-09-28 02:06:50.581 UTC [13856] WARNING: there is no transaction in progress
2026-09-28 02:06:50.595 UTC [13856] WARNING: there is no transaction in progress
2026-09-28 02:06:50.605 UTC [13856] WARNING: there is no transaction in progress
2026-09-28 02:06:50.612 UTC [13856] WARNING: there is no transaction in progress
2026-09-28 02:06:50.620 UTC [13856] WARNING: there is no transaction in progress
2026-09-28 02:06:50.627 UTC [13856] WARNING: there is no transaction in progress
2026-09-28 02:06:51,314 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 02:06:51,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 02:06:51,559 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 02:06:51,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:06:51,804 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 02:06:51,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:06:52,059 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 02:06:52,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:06:52,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:06:52,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-23bp.c.gcp-upstream.internal';"
2026-09-28 02:06:52,739 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 02:06:52,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 02:06:52,988 [I] DB_PATCHES have already been applied
I20260928 02:06:52.994475 1754 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ranger-admin:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasktsFKfV/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 02:06:52.994565 1754 mini_ranger.cc:193] Using host: 127.1.182.148
I20260928 02:06:53.010339 1754 mini_ranger.cc:227] Ranger admin URL: http://127.1.182.148:38567
Sep 28, 2026 2:06:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:06:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:06:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:06:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:06:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:06:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:06:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 02:06:54.226516 1754 mini_ranger.cc:160] Time spent starting Ranger: real 8.606s user 0.002s sys 0.040s
Sep 28, 2026 2:07:13 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:07:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:07:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:07:13 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:07:13 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:07:14 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 02:07:15.176554 1754 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.182.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.182.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.182.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/rangerkms_127.1.182.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.182.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/rangerkms_127.1.182.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.182.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.182.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.182.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/HTTP_127.1.182.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.182.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/HTTP_127.1.182.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 02:07:15 dist-test-slave-23bp krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561235, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 02:07:15.364110 1754 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 02:07:15.384986 1754 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 02:07:16.288973 1754 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 02:07:16.289063 1754 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ranger-kms
I20260928 02:07:16.289197 1754 mini_ranger_kms.cc:227] Using postgres at 127.1.182.148:34387
2026-09-28 02:07:16,655 [I] DB FLAVOR :POSTGRES
2026-09-28 02:07:16,655 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:07:16,655 [I] Checking connection
2026-09-28 02:07:16,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 02:07:16,908 [I] connection success
2026-09-28 02:07:16,908 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 02:07:16,908 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 02:07:16,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 02:07:17,135 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 02:07:17,135 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:07:17,136 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 02:07:17,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.182.148:34387/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 02:07:17,394 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 02:07:17.841008 1754 mini_ranger_kms.cc:310] Created kms service
I20260928 02:07:18.049888 1754 mini_ranger_kms.cc:326] Created kudu user
I20260928 02:07:18.096160 1754 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 02:07:18 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561235, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
I20260928 02:07:18.507344 1754 mini_ranger_kms.cc:384] Added ranger policy
I20260928 02:07:18.507763 1754 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ranger-kms:/tmp/dist-test-tasktsFKfV/build/asan/bin/postgresql.jar:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasktsFKfV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 02:07:18.507828 1754 mini_ranger_kms.cc:240] Using host: 127.1.182.148
I20260928 02:07:18.526126 1754 mini_ranger_kms.cc:276] Ranger KMS PID: 14037
I20260928 02:07:18.526263 1754 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.182.148:55889
Sep 28, 2026 2:07:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:07:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:07:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:07:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:07:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:07:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 02:07:20.934466 1754 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.570s user 0.068s sys 0.317s
I20260928 02:07:20.934746 1754 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 02:07:20.934926 1754 mini_ranger_kms.cc:401] 127.1.182.148:55889/kms/v1/keys
Sep 28 02:07:20 dist-test-slave-23bp krb5kdc[13643](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790561240, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.182.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 02:07:21 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561240, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.182.148@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
Sep 28 02:07:21 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561240, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.182.148@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
Sep 28 02:07:24 dist-test-slave-23bp krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:07:24.226398 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:33515
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:33515
--ranger_config_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.182.148:55889/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:07:24.578473 14088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:24.578800 14088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:24.578965 14088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:24.586117 14088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:07:24.586266 14088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:24.586373 14088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:24.586472 14088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:07:24.586546 14088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:07:24.596683 14088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.182.148:55889/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:33515
--ranger_config_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:33515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:24.597967 14088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:24.599360 14088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:24.611531 14094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:24.611634 14095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:24.611531 14097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:24.612336 14088 server_base.cc:1034] running on GCE node
I20260928 02:07:24.613467 14088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:24.615149 14088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:24.616441 14088 hybrid_clock.cc:648] HybridClock initialized: now 1790561244616337 us; error 51 us; skew 500 ppm
I20260928 02:07:24.616914 14088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:24 dist-test-slave-23bp krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561244, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:24.621464 14088 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:07:24.623518 14088 webserver.cc:460] Webserver started at http://127.1.182.190:38921/ using document root <none> and password file <none>
I20260928 02:07:24.624351 14088 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:24.624574 14088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:07:24.624992 14088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:07:24 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561244, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
I20260928 02:07:24.686627 14088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "2eba3cad751f4dcba8a14a925e7859fe"
format_stamp: "Formatted at 2026-09-28 02:07:24 on dist-test-slave-23bp"
server_key: "15832dae770c2ea2323b6ee5e8b26684"
server_key_iv: "66c7d191063ab84c8a0b61db1933fb81"
server_key_version: "kuduclusterkey@0"
I20260928 02:07:24.687563 14088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "2eba3cad751f4dcba8a14a925e7859fe"
format_stamp: "Formatted at 2026-09-28 02:07:24 on dist-test-slave-23bp"
server_key: "15832dae770c2ea2323b6ee5e8b26684"
server_key_iv: "66c7d191063ab84c8a0b61db1933fb81"
server_key_version: "kuduclusterkey@0"
I20260928 02:07:24.693701 14088 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 02:07:24.710386 14104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:24.712182 14088 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:07:24.712370 14088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "2eba3cad751f4dcba8a14a925e7859fe"
format_stamp: "Formatted at 2026-09-28 02:07:24 on dist-test-slave-23bp"
server_key: "15832dae770c2ea2323b6ee5e8b26684"
server_key_iv: "66c7d191063ab84c8a0b61db1933fb81"
server_key_version: "kuduclusterkey@0"
I20260928 02:07:24.712603 14088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:24.736624 14088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:24.740855 14088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:24.741186 14088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:24.762260 14088 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:33515
I20260928 02:07:24.762265 14155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:33515 every 8 connection(s)
I20260928 02:07:24.763772 14088 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:07:24.767809 14156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:07:24.773149 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 14088
I20260928 02:07:24.773356 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:07:24.774122 1754 external_mini_cluster.cc:1428] Setting key 3fa907845d260488181144cfc2984cae
I20260928 02:07:24.781032 14156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe: Bootstrap starting.
I20260928 02:07:24.786248 14156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe: Neither blocks nor log segments found. Creating new log.
I20260928 02:07:24.788317 14156 log.cc:824] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe: Log is configured to *not* fsync() on all Append() calls
I20260928 02:07:24.795281 14156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe: No bootstrap required, opened a new log
I20260928 02:07:24.809026 14156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eba3cad751f4dcba8a14a925e7859fe" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33515 } }
I20260928 02:07:24.809478 14156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:07:24.809624 14156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eba3cad751f4dcba8a14a925e7859fe, State: Initialized, Role: FOLLOWER
Sep 28 02:07:24 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:07:24.810403 14156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eba3cad751f4dcba8a14a925e7859fe" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33515 } }
I20260928 02:07:24.810714 14156 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:07:24.810868 14156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:07:24.811143 14156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:07:24.814065 14156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eba3cad751f4dcba8a14a925e7859fe" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33515 } }
I20260928 02:07:24.814679 14156 leader_election.cc:302] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2eba3cad751f4dcba8a14a925e7859fe; no voters:
I20260928 02:07:24.817394 14156 leader_election.cc:288] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:07:24.818109 14161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:07:24.820362 14156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:07:24.822609 14161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [term 1 LEADER]: Becoming Leader. State: Replica: 2eba3cad751f4dcba8a14a925e7859fe, State: Running, Role: LEADER
I20260928 02:07:24.823390 14161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eba3cad751f4dcba8a14a925e7859fe" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33515 } }
I20260928 02:07:24.824361 14159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:24.776458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33496 (local address 127.1.182.190:33515)
0928 02:07:24.791662 (+ 15204us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:24.791680 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:24.791744 (+ 64us) server_negotiation.cc:407] Connection header received
0928 02:07:24.795436 (+ 3692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:24.795465 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:24.796085 (+ 620us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:24.796681 (+ 596us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:24.798652 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:24.801313 (+ 2661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:24.803836 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:24.805425 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:24.805613 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:24.811720 (+ 6107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:24.811731 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:24.811751 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:24.811834 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:24.816609 (+ 4775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:24.817558 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:24.817567 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:24.817576 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:24.817764 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:24.818201 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:24.818206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:24.818210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:24.819733 (+ 1523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:24.820099 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:24.821718 (+ 1619us) server_negotiation.cc:299] Negotiation successful
0928 02:07:24.822331 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14833,"thread_start_us":15453,"threads_started":1}
I20260928 02:07:24.834321 14162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2eba3cad751f4dcba8a14a925e7859fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eba3cad751f4dcba8a14a925e7859fe" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33515 } } }
I20260928 02:07:24.834722 14162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [sys.catalog]: This master's current role is: LEADER
I20260928 02:07:24.835238 14163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2eba3cad751f4dcba8a14a925e7859fe. Latest consensus state: current_term: 1 leader_uuid: "2eba3cad751f4dcba8a14a925e7859fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eba3cad751f4dcba8a14a925e7859fe" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33515 } } }
I20260928 02:07:24.835523 14163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [sys.catalog]: This master's current role is: LEADER
I20260928 02:07:24.842833 14171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:07:24.851210 14171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:07:24.865422 14171 catalog_manager.cc:1347] Generated new cluster ID: 456c7cf8f9144d009b2c7c7564c9ba84
I20260928 02:07:24.865593 14171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:07:24.910624 14156 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:07:24.911384 14171 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:07:24.913946 14171 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:07:24.936794 14171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe: Generated new TSK 0
I20260928 02:07:24.938426 14171 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.182.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
I20260928 02:07:25.690840 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.182.148:55889/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:33515
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:07:25 dist-test-slave-23bp krb5kdc[13643](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790561245, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 02:07:26.108597 14203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:26.109102 14203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:26.109309 14203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:26.120965 14203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:26.121101 14203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:26.121337 14203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:07:26.137048 14203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.182.148:55889/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:33515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:26.138713 14203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:26.140435 14203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:26.169354 14213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:26.181849 14214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:26.182161 14216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:26.182967 14203 server_base.cc:1034] running on GCE node
I20260928 02:07:26.183983 14203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:26.185808 14203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:26.187153 14203 hybrid_clock.cc:648] HybridClock initialized: now 1790561246187080 us; error 58 us; skew 500 ppm
I20260928 02:07:26.187554 14203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:26 dist-test-slave-23bp krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561246, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:26.191334 14203 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:07:26.193101 14203 webserver.cc:460] Webserver started at http://127.1.182.129:37621/ using document root <none> and password file <none>
I20260928 02:07:26.194062 14203 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:26.194190 14203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:07:26.194505 14203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:07:26 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561246, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
I20260928 02:07:26.214360 14203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "2055bca407f04439a1fb9b08717157b7"
format_stamp: "Formatted at 2026-09-28 02:07:26 on dist-test-slave-23bp"
server_key: "391805681e127223120a61c45e7b6b06"
server_key_iv: "5a210ae3e3f5971992f64a9a21693cb6"
server_key_version: "kuduclusterkey@0"
I20260928 02:07:26.215701 14203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "2055bca407f04439a1fb9b08717157b7"
format_stamp: "Formatted at 2026-09-28 02:07:26 on dist-test-slave-23bp"
server_key: "391805681e127223120a61c45e7b6b06"
server_key_iv: "5a210ae3e3f5971992f64a9a21693cb6"
server_key_version: "kuduclusterkey@0"
I20260928 02:07:26.234349 14203 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.005s sys 0.012s
I20260928 02:07:26.260532 14223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:26.264397 14203 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.001s sys 0.008s
I20260928 02:07:26.264622 14203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "2055bca407f04439a1fb9b08717157b7"
format_stamp: "Formatted at 2026-09-28 02:07:26 on dist-test-slave-23bp"
server_key: "391805681e127223120a61c45e7b6b06"
server_key_iv: "5a210ae3e3f5971992f64a9a21693cb6"
server_key_version: "kuduclusterkey@0"
I20260928 02:07:26.264871 14203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:26.284849 14203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:26.289456 14203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:26.289817 14203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:26.290866 14203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:07:26.294019 14203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:07:26.294173 14203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 02:07:26.294374 14203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:07:26.294456 14203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:26.363430 14203 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:38931
I20260928 02:07:26.363481 14336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:38931 every 8 connection(s)
I20260928 02:07:26.365195 14203 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:07:26.373881 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 14203
I20260928 02:07:26.374158 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:07:26.375216 1754 external_mini_cluster.cc:1428] Setting key 13322f423438580938204bee7451412c
Sep 28 02:07:26 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561246, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:07:26.400085 14339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:26.369961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:52539 (local address 127.1.182.190:33515)
0928 02:07:26.377688 (+ 7727us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:26.377696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:26.378395 (+ 699us) server_negotiation.cc:407] Connection header received
0928 02:07:26.381009 (+ 2614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:26.381014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:26.381110 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:26.381195 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:26.383522 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:26.384781 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:26.387422 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:26.388156 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:26.388267 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:26.392891 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:26.392898 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:26.392904 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:26.392946 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:26.396139 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:26.397041 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:26.397049 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:26.397056 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:26.397210 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:26.397719 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:26.397727 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:26.397733 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:26.398109 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:26.398271 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:26.399326 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0928 02:07:26.399575 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7571,"thread_start_us":8723,"threads_started":1}
I20260928 02:07:26.402386 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:26.370446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33515 (local address 127.1.182.129:52539)
0928 02:07:26.378226 (+ 7780us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:26.378276 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:26.380676 (+ 2400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:26.381808 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:26.381822 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:26.382473 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:26.383324 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:26.383352 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:26.384989 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:26.384997 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:26.387248 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:26.387275 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:26.388640 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:26.388649 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:26.388972 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:26.389788 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:26.389818 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:26.392618 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:26.396282 (+ 3664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:26.396294 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:26.396316 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:26.396903 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:26.397348 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:26.397355 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:26.397362 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:26.397593 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:26.398361 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:26.398372 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:26.398838 (+ 466us) client_negotiation.cc:769] Sending connection context
0928 02:07:26.400334 (+ 1496us) client_negotiation.cc:240] Negotiation successful
0928 02:07:26.400905 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7482,"thread_start_us":7258,"threads_started":1}
I20260928 02:07:26.404399 14337 heartbeater.cc:347] Connected to a master server at 127.1.182.190:33515
I20260928 02:07:26.404702 14337 heartbeater.cc:464] Registering TS with master...
I20260928 02:07:26.405534 14337 heartbeater.cc:511] Master 127.1.182.190:33515 requested a full tablet report, sending...
I20260928 02:07:26.408118 14121 ts_manager.cc:194] Registered new tserver with Master: 2055bca407f04439a1fb9b08717157b7 (127.1.182.129:38931)
I20260928 02:07:26.410231 14121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:52539
Sep 28 02:07:26 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561245, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.182.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
I20260928 02:07:26.451398 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.182.148:55889/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:33515
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:07:26 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561245, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
W20260928 02:07:26.892498 14343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:26.893003 14343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:26.893193 14343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:26.905789 14343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:26.906032 14343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:26.906283 14343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:07:26.923101 14343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.182.148:55889/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:33515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:26.925057 14343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:26.926784 14343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:26.962323 14350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:26.963974 14351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:26.966799 14343 server_base.cc:1034] running on GCE node
W20260928 02:07:26.967216 14353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:26.968119 14343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:26.971375 14343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:26.972731 14343 hybrid_clock.cc:648] HybridClock initialized: now 1790561246972603 us; error 66 us; skew 500 ppm
I20260928 02:07:26.973136 14343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:26 dist-test-slave-23bp krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561246, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:26.976931 14343 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:07:26.980491 14343 webserver.cc:460] Webserver started at http://127.1.182.130:38133/ using document root <none> and password file <none>
I20260928 02:07:26.981281 14343 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:26.981444 14343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:07:26.981787 14343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:07:26 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561246, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
I20260928 02:07:27.001083 14343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "ac84e640a1094fd89c94dfa8146ac3e8"
format_stamp: "Formatted at 2026-09-28 02:07:26 on dist-test-slave-23bp"
server_key: "ff7069ad2e411ff13da0fc68d1cd7c6a"
server_key_iv: "6681e2fb5a0ee6c01f02708454efa8d3"
server_key_version: "kuduclusterkey@0"
I20260928 02:07:27.002249 14343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "ac84e640a1094fd89c94dfa8146ac3e8"
format_stamp: "Formatted at 2026-09-28 02:07:26 on dist-test-slave-23bp"
server_key: "ff7069ad2e411ff13da0fc68d1cd7c6a"
server_key_iv: "6681e2fb5a0ee6c01f02708454efa8d3"
server_key_version: "kuduclusterkey@0"
I20260928 02:07:27.013326 14343 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.000s sys 0.006s
I20260928 02:07:27.037964 14367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:27.039870 14343 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:07:27.040071 14343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "ac84e640a1094fd89c94dfa8146ac3e8"
format_stamp: "Formatted at 2026-09-28 02:07:26 on dist-test-slave-23bp"
server_key: "ff7069ad2e411ff13da0fc68d1cd7c6a"
server_key_iv: "6681e2fb5a0ee6c01f02708454efa8d3"
server_key_version: "kuduclusterkey@0"
I20260928 02:07:27.040338 14343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:27.051532 14343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:27.055507 14343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:27.055799 14343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:27.056770 14343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:07:27.058226 14343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:07:27.058296 14343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:27.058419 14343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:07:27.058465 14343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:27.094128 14343 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:42201
I20260928 02:07:27.094177 14480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:42201 every 8 connection(s)
I20260928 02:07:27.095496 14343 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:07:27.105513 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 14343
I20260928 02:07:27.105746 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:07:27.106297 1754 external_mini_cluster.cc:1428] Setting key d55a4387046b35db178ad642fbe75640
Sep 28 02:07:27 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561246, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:07:27.116264 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:27.097859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:57159 (local address 127.1.182.190:33515)
0928 02:07:27.098316 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:27.098323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:27.099306 (+ 983us) server_negotiation.cc:407] Connection header received
0928 02:07:27.100980 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:27.100985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:27.101061 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:27.101164 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:27.102472 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:27.103355 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:27.104808 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:27.105303 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:27.105435 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:27.109108 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:27.109117 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:27.109125 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:27.109176 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:27.112856 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:27.113563 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:27.113571 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:27.113577 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:27.113738 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:27.114160 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:27.114167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:27.114173 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:27.114553 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:27.114708 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:27.115487 (+ 779us) server_negotiation.cc:299] Negotiation successful
0928 02:07:27.115751 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":207,"threads_started":1}
I20260928 02:07:27.117062 14484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:27.098434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33515 (local address 127.1.182.130:57159)
0928 02:07:27.099165 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:27.099202 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:27.100739 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:27.101353 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:27.101360 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:27.101774 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:27.102317 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:27.102333 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:27.103476 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:27.103480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:27.104686 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:27.104704 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:27.105506 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:27.105511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:27.105810 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:27.106548 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:27.106578 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:27.108886 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:27.113006 (+ 4120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:27.113013 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:27.113027 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:27.113438 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:27.113848 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:27.113851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:27.113853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:27.114045 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:27.114670 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:27.114676 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:27.115054 (+ 378us) client_negotiation.cc:769] Sending connection context
0928 02:07:27.115372 (+ 318us) client_negotiation.cc:240] Negotiation successful
0928 02:07:27.115757 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":270,"threads_started":1}
I20260928 02:07:27.118487 14481 heartbeater.cc:347] Connected to a master server at 127.1.182.190:33515
I20260928 02:07:27.118798 14481 heartbeater.cc:464] Registering TS with master...
I20260928 02:07:27.119647 14481 heartbeater.cc:511] Master 127.1.182.190:33515 requested a full tablet report, sending...
I20260928 02:07:27.121157 14121 ts_manager.cc:194] Registered new tserver with Master: ac84e640a1094fd89c94dfa8146ac3e8 (127.1.182.130:42201)
I20260928 02:07:27.122092 14121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:57159
WARNING: no policy specified for kudu/127.1.182.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
I20260928 02:07:27.158769 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.182.148:55889/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:33515
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:07:27.414204 14337 heartbeater.cc:503] Master 127.1.182.190:33515 was elected leader, sending a full tablet report...
W20260928 02:07:27.484408 14487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:27.484665 14487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:27.484752 14487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:27.491775 14487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:27.491870 14487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:27.491976 14487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:07:27.501387 14487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.182.148:55889/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:33515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:27.502569 14487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:27.503773 14487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:27.516753 14492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:27.516791 14493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:27.517122 14495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:27.517720 14487 server_base.cc:1034] running on GCE node
I20260928 02:07:27.518324 14487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:27.519321 14487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:27.520507 14487 hybrid_clock.cc:648] HybridClock initialized: now 1790561247520473 us; error 31 us; skew 500 ppm
I20260928 02:07:27.520829 14487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:27 dist-test-slave-23bp krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561247, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:27.524006 14487 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:07:27.525363 14487 webserver.cc:460] Webserver started at http://127.1.182.131:44791/ using document root <none> and password file <none>
I20260928 02:07:27.526031 14487 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:27.526157 14487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:07:27.526463 14487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:07:27 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561247, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
I20260928 02:07:27.543990 14487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "bdac00a0660849e6b5b2437be1385a81"
format_stamp: "Formatted at 2026-09-28 02:07:27 on dist-test-slave-23bp"
server_key: "0e617b61abb18cf6d637c370001aa330"
server_key_iv: "1463edd2aa4b9649005590b7eaeb682b"
server_key_version: "kuduclusterkey@0"
I20260928 02:07:27.545137 14487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "bdac00a0660849e6b5b2437be1385a81"
format_stamp: "Formatted at 2026-09-28 02:07:27 on dist-test-slave-23bp"
server_key: "0e617b61abb18cf6d637c370001aa330"
server_key_iv: "1463edd2aa4b9649005590b7eaeb682b"
server_key_version: "kuduclusterkey@0"
I20260928 02:07:27.552208 14487 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260928 02:07:27.570284 14502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:27.572124 14487 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 02:07:27.572312 14487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "bdac00a0660849e6b5b2437be1385a81"
format_stamp: "Formatted at 2026-09-28 02:07:27 on dist-test-slave-23bp"
server_key: "0e617b61abb18cf6d637c370001aa330"
server_key_iv: "1463edd2aa4b9649005590b7eaeb682b"
server_key_version: "kuduclusterkey@0"
I20260928 02:07:27.572589 14487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:27.592417 14487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:27.597472 14487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:27.597808 14487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:27.598912 14487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:07:27.600764 14487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:07:27.600878 14487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:27.601011 14487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:07:27.601095 14487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:27.643347 14487 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:33597
I20260928 02:07:27.643368 14614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:33597 every 8 connection(s)
I20260928 02:07:27.645089 14487 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:07:27.651860 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 14487
I20260928 02:07:27.652034 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:07:27.652706 1754 external_mini_cluster.cc:1428] Setting key 244b514b819ba6dcfc1de95a2a30891a
Sep 28 02:07:27 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561247, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:07:27.666143 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:27.647867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:39689 (local address 127.1.182.190:33515)
0928 02:07:27.648466 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:27.648471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:27.649438 (+ 967us) server_negotiation.cc:407] Connection header received
0928 02:07:27.651200 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:27.651206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:27.651294 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:27.651414 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:27.652975 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:27.653896 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:27.656206 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:27.656610 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:27.656703 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:27.660475 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:27.660480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:27.660483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:27.660514 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:27.663160 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:27.663871 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:27.663877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:27.663882 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:27.664005 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:27.664434 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:27.664439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:27.664443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:27.664708 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:27.664861 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:27.665519 (+ 658us) server_negotiation.cc:299] Negotiation successful
0928 02:07:27.665734 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":316,"threads_started":1}
I20260928 02:07:27.667388 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:27.648530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33515 (local address 127.1.182.131:39689)
0928 02:07:27.649284 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:27.649322 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:27.650881 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:27.651656 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:27.651665 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:27.652202 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:27.652794 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:27.652819 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:27.654066 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:27.654072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:27.656057 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:27.656080 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:27.656846 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:27.656853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:27.657134 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:27.657845 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:27.657873 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:27.660267 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:27.663295 (+ 3028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:27.663303 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:27.663319 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:27.663743 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:27.664111 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:27.664119 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:27.664124 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:27.664316 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:27.664833 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:27.664842 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:27.665167 (+ 325us) client_negotiation.cc:769] Sending connection context
0928 02:07:27.665424 (+ 257us) client_negotiation.cc:240] Negotiation successful
0928 02:07:27.665940 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":316,"threads_started":1}
I20260928 02:07:27.669037 14615 heartbeater.cc:347] Connected to a master server at 127.1.182.190:33515
I20260928 02:07:27.669339 14615 heartbeater.cc:464] Registering TS with master...
I20260928 02:07:27.670104 14615 heartbeater.cc:511] Master 127.1.182.190:33515 requested a full tablet report, sending...
I20260928 02:07:27.671694 14120 ts_manager.cc:194] Registered new tserver with Master: bdac00a0660849e6b5b2437be1385a81 (127.1.182.131:33597)
I20260928 02:07:27.672992 14120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:39689
I20260928 02:07:27.681638 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:07:27.684135 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 14203
Sep 28 02:07:27 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:07:27.701259 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:27.685215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34226 (local address 127.1.182.129:38931)
0928 02:07:27.685903 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:27.685910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:27.685933 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:07:27.685989 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:27.685995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:27.686215 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:27.686392 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:27.688027 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:27.689028 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:27.690476 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:27.691166 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:27.691305 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:27.695512 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:27.695523 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:27.695537 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:27.695569 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:27.698269 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:27.699002 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:27.699009 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:27.699012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:27.699247 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:27.699640 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:27.699647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:27.699650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:27.699984 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:27.700309 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:27.700628 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 02:07:27.700897 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":327,"threads_started":1}
I20260928 02:07:27.702491 14332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:07:27.812438 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 14203
I20260928 02:07:27.829041 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 14343
Sep 28 02:07:27 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.130@KRBTEST.COM
I20260928 02:07:27.845736 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:27.829990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47164 (local address 127.1.182.130:42201)
0928 02:07:27.830534 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:27.830540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:27.830564 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:07:27.830630 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:27.830636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:27.830838 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:27.831033 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:27.832345 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:27.833330 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:27.834834 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:27.835460 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:27.835602 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:27.839785 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:27.839795 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:27.839809 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:27.839841 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:27.842547 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:27.843504 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:27.843537 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:27.843540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:27.843699 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:27.844150 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:27.844153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:27.844155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:27.844479 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:27.844759 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:27.845153 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 02:07:27.845350 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":292,"threads_started":1}
I20260928 02:07:27.846949 14476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:07:27.930667 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 14343
I20260928 02:07:27.947469 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 14487
Sep 28 02:07:27 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.131@KRBTEST.COM
I20260928 02:07:27.968516 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:27.948719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49536 (local address 127.1.182.131:33597)
0928 02:07:27.949362 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:27.949370 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:27.949399 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:07:27.949458 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:27.949465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:27.949693 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:27.949820 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:27.951703 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:27.952695 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:27.954602 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:27.955257 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:27.955353 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:27.962176 (+ 6823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:27.962195 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:27.962221 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:27.962266 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:27.965500 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:27.966272 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:27.966280 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:27.966283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:27.966414 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:27.966821 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:27.966827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:27.966830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:27.967251 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:27.967568 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:27.967877 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 02:07:27.968139 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":351,"threads_started":1}
I20260928 02:07:27.969794 14610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:07:28.055938 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 14487
I20260928 02:07:28.071736 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 14088
I20260928 02:07:28.072969 14151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:07:28.178332 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 14088
I20260928 02:07:28.199316 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:33515
--webserver_interface=127.1.182.190
--webserver_port=38921
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:33515
--ranger_config_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.182.148:55889/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.182.148:55889/kms with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:07:28.523048 14627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:28.523422 14627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:28.523595 14627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:28.530926 14627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:07:28.531078 14627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:28.531232 14627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:28.531337 14627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:07:28.531418 14627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:07:28.541692 14627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.182.148:55889/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:33515
--ranger_config_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:33515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=38921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:28.543293 14627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:28.544709 14627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:28.556977 14632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:28.556943 14634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:28.557036 14636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:28.558379 14627 server_base.cc:1034] running on GCE node
I20260928 02:07:28.559275 14627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:28.560709 14627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:28.561937 14627 hybrid_clock.cc:648] HybridClock initialized: now 1790561248561897 us; error 56 us; skew 500 ppm
I20260928 02:07:28.562285 14627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:28 dist-test-slave-23bp krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561248, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:28.569379 14627 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:07:28.570981 14627 webserver.cc:460] Webserver started at http://127.1.182.190:38921/ using document root <none> and password file <none>
I20260928 02:07:28.571697 14627 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:28.571846 14627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:07:28 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561248, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
I20260928 02:07:28.626022 14627 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.000s sys 0.007s
I20260928 02:07:28.629241 14644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:28.630671 14627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:07:28.630842 14627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "2eba3cad751f4dcba8a14a925e7859fe"
format_stamp: "Formatted at 2026-09-28 02:07:24 on dist-test-slave-23bp"
server_key: "15832dae770c2ea2323b6ee5e8b26684"
server_key_iv: "66c7d191063ab84c8a0b61db1933fb81"
server_key_version: "kuduclusterkey@0"
I20260928 02:07:28.631903 14627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:28.651559 14627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:28.655220 14627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:28.655500 14627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:28.673336 14627 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:33515
I20260928 02:07:28.673357 14695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:33515 every 8 connection(s)
I20260928 02:07:28.674844 14627 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:07:28.681568 14696 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:07:28.681802 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 14627
I20260928 02:07:28.682765 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.182.148:55889/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:38931
--local_ip_for_outbound_sockets=127.1.182.129
--tserver_master_addrs=127.1.182.190:33515
--webserver_port=37621
--webserver_interface=127.1.182.129
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.182.148:55889/kms with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:07:28.685014 14696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe: Bootstrap starting.
I20260928 02:07:28.703702 14696 log.cc:824] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe: Log is configured to *not* fsync() on all Append() calls
I20260928 02:07:28.712275 14696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:07:28.712777 14696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe: Bootstrap complete.
I20260928 02:07:28.721534 14696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eba3cad751f4dcba8a14a925e7859fe" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33515 } }
I20260928 02:07:28.722376 14696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eba3cad751f4dcba8a14a925e7859fe, State: Initialized, Role: FOLLOWER
I20260928 02:07:28.723006 14696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eba3cad751f4dcba8a14a925e7859fe" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33515 } }
I20260928 02:07:28.723299 14696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:07:28.723420 14696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:07:28.723608 14696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:07:28.727284 14696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eba3cad751f4dcba8a14a925e7859fe" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33515 } }
I20260928 02:07:28.727738 14696 leader_election.cc:302] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2eba3cad751f4dcba8a14a925e7859fe; no voters:
I20260928 02:07:28.728368 14696 leader_election.cc:288] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:07:28.728506 14700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:07:28.729622 14700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [term 2 LEADER]: Becoming Leader. State: Replica: 2eba3cad751f4dcba8a14a925e7859fe, State: Running, Role: LEADER
I20260928 02:07:28.730217 14700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eba3cad751f4dcba8a14a925e7859fe" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33515 } }
I20260928 02:07:28.730572 14696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:07:28.732771 14701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2eba3cad751f4dcba8a14a925e7859fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eba3cad751f4dcba8a14a925e7859fe" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33515 } } }
I20260928 02:07:28.733197 14701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [sys.catalog]: This master's current role is: LEADER
I20260928 02:07:28.734814 14705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:07:28.735157 14702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2eba3cad751f4dcba8a14a925e7859fe. Latest consensus state: current_term: 2 leader_uuid: "2eba3cad751f4dcba8a14a925e7859fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eba3cad751f4dcba8a14a925e7859fe" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 33515 } } }
I20260928 02:07:28.735520 14702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe [sys.catalog]: This master's current role is: LEADER
I20260928 02:07:28.741616 14705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:07:28.742995 14705 catalog_manager.cc:1259] Loaded cluster ID: 456c7cf8f9144d009b2c7c7564c9ba84
I20260928 02:07:28.743149 14705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:07:28.747041 14705 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:07:28.748567 14705 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe: Loaded TSK: 0
I20260928 02:07:28.750557 14705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:07:28.757066 14696 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260928 02:07:29.105883 14698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:29.106335 14698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:29.106529 14698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:29.118366 14698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:29.118507 14698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:29.118789 14698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:07:29.134634 14698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.182.148:55889/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:38931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=37621
--tserver_master_addrs=127.1.182.190:33515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:29.136562 14698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:29.138373 14698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:29.166837 14733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:29.168578 14734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:29.179343 14698 server_base.cc:1034] running on GCE node
W20260928 02:07:29.180788 14736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:29.181736 14698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:29.187527 14698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:29.188856 14698 hybrid_clock.cc:648] HybridClock initialized: now 1790561249188775 us; error 81 us; skew 500 ppm
I20260928 02:07:29.189343 14698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:29 dist-test-slave-23bp krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:29.193950 14698 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:07:29.201741 14698 webserver.cc:460] Webserver started at http://127.1.182.129:37621/ using document root <none> and password file <none>
I20260928 02:07:29.202867 14698 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:29.203161 14698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:07:29 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
I20260928 02:07:29.252620 14698 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.009s sys 0.001s
I20260928 02:07:29.255798 14744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:29.257344 14698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:07:29.257536 14698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "2055bca407f04439a1fb9b08717157b7"
format_stamp: "Formatted at 2026-09-28 02:07:26 on dist-test-slave-23bp"
server_key: "391805681e127223120a61c45e7b6b06"
server_key_iv: "5a210ae3e3f5971992f64a9a21693cb6"
server_key_version: "kuduclusterkey@0"
I20260928 02:07:29.258884 14698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:29.316852 14698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:29.321790 14698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:29.322121 14698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:29.323341 14698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:07:29.324958 14698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:07:29.325093 14698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:29.325232 14698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:07:29.325341 14698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:29.404693 14698 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:38931
I20260928 02:07:29.404780 14858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:38931 every 8 connection(s)
I20260928 02:07:29.407032 14698 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:07:29.407708 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 14698
I20260928 02:07:29.408514 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.182.148:55889/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:42201
--local_ip_for_outbound_sockets=127.1.182.130
--tserver_master_addrs=127.1.182.190:33515
--webserver_port=38133
--webserver_interface=127.1.182.130
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.182.148:55889/kms with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:07:29 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:07:29.451890 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:29.417734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:36841 (local address 127.1.182.190:33515)
0928 02:07:29.426327 (+ 8593us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:29.426353 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:29.429395 (+ 3042us) server_negotiation.cc:407] Connection header received
0928 02:07:29.429882 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:29.429902 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:29.430410 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:29.430872 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:29.433444 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:29.435529 (+ 2085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:29.438392 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:29.439485 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:29.439673 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:29.443841 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:29.443848 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:29.443863 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:29.443902 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:29.446859 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:29.447711 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:29.447720 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:29.447728 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:29.447938 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:29.448421 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:29.448426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:29.448429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:29.448926 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:29.449137 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:29.449930 (+ 793us) server_negotiation.cc:299] Negotiation successful
0928 02:07:29.450297 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8299,"thread_start_us":8059,"threads_started":1}
I20260928 02:07:29.452740 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:29.417118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33515 (local address 127.1.182.129:36841)
0928 02:07:29.426763 (+ 9645us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:29.426823 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:29.428966 (+ 2143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:29.431229 (+ 2263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:29.431242 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:29.431957 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:29.432830 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:29.432856 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:29.435763 (+ 2907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:29.435769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:29.438176 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:29.438191 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:29.440063 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:29.440071 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:29.440356 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:29.441188 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:29.441220 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:29.443503 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:29.447013 (+ 3510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:29.447023 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:29.447039 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:29.447549 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:29.448072 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:29.448078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:29.448082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:29.448295 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:29.449081 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:29.449090 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:29.449437 (+ 347us) client_negotiation.cc:769] Sending connection context
0928 02:07:29.449702 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 02:07:29.450281 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9324,"thread_start_us":9144,"threads_started":1}
I20260928 02:07:29.456769 14859 heartbeater.cc:347] Connected to a master server at 127.1.182.190:33515
I20260928 02:07:29.457206 14859 heartbeater.cc:464] Registering TS with master...
I20260928 02:07:29.457960 14859 heartbeater.cc:511] Master 127.1.182.190:33515 requested a full tablet report, sending...
W20260928 02:07:29.460913 14859 heartbeater.cc:644] Failed to heartbeat to 127.1.182.190:33515 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:07:29.461156 14859 heartbeater.cc:464] Registering TS with master...
I20260928 02:07:29.461419 14859 heartbeater.cc:511] Master 127.1.182.190:33515 requested a full tablet report, sending...
I20260928 02:07:29.462581 14859 heartbeater.cc:464] Registering TS with master...
I20260928 02:07:29.462843 14859 heartbeater.cc:511] Master 127.1.182.190:33515 requested a full tablet report, sending...
W20260928 02:07:29.463943 14859 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:07:29.837229 14861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:29.837602 14861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:29.837760 14861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:29.848869 14861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:29.849033 14861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:29.849233 14861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:07:29.862931 14861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.182.148:55889/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:42201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=38133
--tserver_master_addrs=127.1.182.190:33515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:29.864533 14861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:29.866266 14861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:29.883452 14878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:29.885809 14877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:29.887303 14881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:29.891301 14861 server_base.cc:1034] running on GCE node
I20260928 02:07:29.892215 14861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:29.902436 14861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:29.903748 14861 hybrid_clock.cc:648] HybridClock initialized: now 1790561249903681 us; error 77 us; skew 500 ppm
I20260928 02:07:29.904225 14861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:29 dist-test-slave-23bp krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:29.908632 14861 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:07:29.927196 14861 webserver.cc:460] Webserver started at http://127.1.182.130:38133/ using document root <none> and password file <none>
I20260928 02:07:29.928511 14861 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:29.928756 14861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:07:29 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
I20260928 02:07:29.962725 14861 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.000s sys 0.007s
I20260928 02:07:29.966543 14891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:29.968385 14861 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 02:07:29.968627 14861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "ac84e640a1094fd89c94dfa8146ac3e8"
format_stamp: "Formatted at 2026-09-28 02:07:26 on dist-test-slave-23bp"
server_key: "ff7069ad2e411ff13da0fc68d1cd7c6a"
server_key_iv: "6681e2fb5a0ee6c01f02708454efa8d3"
server_key_version: "kuduclusterkey@0"
I20260928 02:07:29.970247 14861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 28 02:07:29 dist-test-slave-23bp krb5kdc[13643](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790561249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:29.991688 14861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:29.999125 14861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:29.999634 14861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:30.001875 14861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:07:30.007954 14861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:07:30.008147 14861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:30.008327 14861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:07:30.008446 14861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:30.083077 14861 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:42201
I20260928 02:07:30.083156 15004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:42201 every 8 connection(s)
I20260928 02:07:30.084781 14861 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:07:30.091161 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 14861
I20260928 02:07:30.091939 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.182.148:55889/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:33597
--local_ip_for_outbound_sockets=127.1.182.131
--tserver_master_addrs=127.1.182.190:33515
--webserver_port=44791
--webserver_interface=127.1.182.131
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.182.148:55889/kms with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:07:30 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:07:30.123826 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:30.088196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:37957 (local address 127.1.182.190:33515)
0928 02:07:30.099782 (+ 11586us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:30.099791 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:30.100651 (+ 860us) server_negotiation.cc:407] Connection header received
0928 02:07:30.103756 (+ 3105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:30.103764 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:30.103854 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:30.103980 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:30.106171 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:30.107535 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:30.109990 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:30.110823 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:30.110971 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:30.115517 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:30.115527 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:30.115535 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:30.115587 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:30.119769 (+ 4182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:30.120769 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:30.120774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:30.120778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:30.120914 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:30.121407 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:30.121412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:30.121415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:30.121755 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:30.122005 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:30.123226 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0928 02:07:30.123442 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11416,"thread_start_us":11313,"threads_started":1}
I20260928 02:07:30.126067 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:30.088875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33515 (local address 127.1.182.130:37957)
0928 02:07:30.100438 (+ 11563us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:30.100517 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:30.103411 (+ 2894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:30.104345 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:30.104362 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:30.105094 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:30.105963 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:30.105990 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:30.107723 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:30.107728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:30.109785 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:30.109799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:30.111119 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:30.111128 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:30.111415 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:30.112295 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:30.112333 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:30.115240 (+ 2907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:30.119964 (+ 4724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:30.119976 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:30.119998 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:30.120609 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:30.121033 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:30.121038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:30.121042 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:30.121282 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:30.122075 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:30.122083 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:30.122516 (+ 433us) client_negotiation.cc:769] Sending connection context
0928 02:07:30.122817 (+ 301us) client_negotiation.cc:240] Negotiation successful
0928 02:07:30.124678 (+ 1861us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11200,"thread_start_us":10932,"threads_started":1}
I20260928 02:07:30.128130 15005 heartbeater.cc:347] Connected to a master server at 127.1.182.190:33515
I20260928 02:07:30.128446 15005 heartbeater.cc:464] Registering TS with master...
I20260928 02:07:30.129263 15005 heartbeater.cc:511] Master 127.1.182.190:33515 requested a full tablet report, sending...
I20260928 02:07:30.131381 14661 ts_manager.cc:194] Registered new tserver with Master: ac84e640a1094fd89c94dfa8146ac3e8 (127.1.182.130:42201)
I20260928 02:07:30.133535 14661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:37957
Sep 28 02:07:30 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
I20260928 02:07:30.464078 14859 heartbeater.cc:464] Registering TS with master...
I20260928 02:07:30.464568 14859 heartbeater.cc:511] Master 127.1.182.190:33515 requested a full tablet report, sending...
I20260928 02:07:30.465960 14661 ts_manager.cc:194] Registered new tserver with Master: 2055bca407f04439a1fb9b08717157b7 (127.1.182.129:38931)
I20260928 02:07:30.467262 14661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:36841
W20260928 02:07:30.576095 15009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:30.576663 15009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:30.576925 15009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:30.589506 15009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:30.589732 15009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:30.589974 15009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:07:30.605581 15009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.182.148:55889/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:33597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=44791
--tserver_master_addrs=127.1.182.190:33515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:30.607609 15009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:30.609449 15009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:30.647614 15015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:30.648514 15018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:30.649830 15014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:30.651086 15009 server_base.cc:1034] running on GCE node
I20260928 02:07:30.652063 15009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:30.653568 15009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:30.654891 15009 hybrid_clock.cc:648] HybridClock initialized: now 1790561250654820 us; error 55 us; skew 500 ppm
I20260928 02:07:30.655390 15009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:30 dist-test-slave-23bp krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561250, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:30.659765 15009 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:07:30.661875 15009 webserver.cc:460] Webserver started at http://127.1.182.131:44791/ using document root <none> and password file <none>
I20260928 02:07:30.662827 15009 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:30.662974 15009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:07:30 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561250, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
I20260928 02:07:30.691989 15009 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.004s
I20260928 02:07:30.695706 15025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:30.697664 15009 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:07:30.697852 15009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "bdac00a0660849e6b5b2437be1385a81"
format_stamp: "Formatted at 2026-09-28 02:07:27 on dist-test-slave-23bp"
server_key: "0e617b61abb18cf6d637c370001aa330"
server_key_iv: "1463edd2aa4b9649005590b7eaeb682b"
server_key_version: "kuduclusterkey@0"
I20260928 02:07:30.699333 15009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:30.723449 15009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:30.728636 15009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:30.728927 15009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:30.730055 15009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:07:30.734359 15009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:07:30.734540 15009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 02:07:30.734709 15009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:07:30.734840 15009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:07:30 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for HTTP/127.1.182.148@KRBTEST.COM
I20260928 02:07:31.048264 15140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:33597 every 8 connection(s)
I20260928 02:07:31.048604 15009 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:33597
I20260928 02:07:31.050590 15009 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:07:31.051093 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 15009
Sep 28 02:07:31 dist-test-slave-23bp krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561250, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:07:31.106752 15143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.053033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60578 (local address 127.1.182.190:33515)
0928 02:07:31.072497 (+ 19464us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:31.072504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:31.072526 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:07:31.072603 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:31.072608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:31.072698 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:31.072811 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:31.081441 (+ 8630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.082597 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.084756 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.085398 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.085523 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:31.095269 (+ 9746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:31.095276 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:31.095281 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:31.095361 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:31.099520 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:31.100528 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:31.100550 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:31.100555 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:31.100706 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:31.101249 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:31.101255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:31.101258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:31.101633 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:31.101884 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:31.106087 (+ 4203us) server_negotiation.cc:299] Negotiation successful
0928 02:07:31.106316 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19317,"spinlock_wait_cycles":10112,"thread_start_us":19881,"threads_started":1}
I20260928 02:07:31.111090 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.058549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:59649 (local address 127.1.182.190:33515)
0928 02:07:31.084654 (+ 26105us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:31.084660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:31.087116 (+ 2456us) server_negotiation.cc:407] Connection header received
0928 02:07:31.088826 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:31.088831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:31.088934 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:31.089020 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:31.091193 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.092280 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.094494 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.095169 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.095261 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:31.102232 (+ 6971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:31.102239 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:31.102244 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:31.102280 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:31.105621 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:31.107632 (+ 2011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:31.107636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:31.107639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:31.107770 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:31.108675 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:31.108682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:31.108688 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:31.109023 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:31.109172 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:31.110464 (+ 1292us) server_negotiation.cc:299] Negotiation successful
0928 02:07:31.110694 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25964,"thread_start_us":26117,"threads_started":1}
I20260928 02:07:31.113762 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.058023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:33515 (local address 127.1.182.131:59649)
0928 02:07:31.085274 (+ 27251us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:31.085323 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:31.087733 (+ 2410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:31.089579 (+ 1846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:31.089592 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:31.090180 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:31.090920 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:31.090941 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.092458 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.092464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:31.094341 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:31.094355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.095606 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.095612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:31.095880 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:31.099211 (+ 3331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:31.099241 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:31.101917 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:31.106689 (+ 4772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:31.106698 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:31.106720 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:31.107380 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:31.108058 (+ 678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:31.108063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:31.108066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:31.108533 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:31.109271 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:31.109279 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:31.110014 (+ 735us) client_negotiation.cc:769] Sending connection context
0928 02:07:31.111691 (+ 1677us) client_negotiation.cc:240] Negotiation successful
0928 02:07:31.112300 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26914,"thread_start_us":29905,"threads_started":1}
I20260928 02:07:31.115710 15141 heartbeater.cc:347] Connected to a master server at 127.1.182.190:33515
I20260928 02:07:31.116016 15141 heartbeater.cc:464] Registering TS with master...
I20260928 02:07:31.116889 15141 heartbeater.cc:511] Master 127.1.182.190:33515 requested a full tablet report, sending...
I20260928 02:07:31.118876 14661 ts_manager.cc:194] Registered new tserver with Master: bdac00a0660849e6b5b2437be1385a81 (127.1.182.131:33597)
I20260928 02:07:31.120071 14661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:59649
I20260928 02:07:31.125214 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:07:31.138027 15005 heartbeater.cc:503] Master 127.1.182.190:33515 was elected leader, sending a full tablet report...
I20260928 02:07:31.175166 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.156240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.1.182.190:33515)
0928 02:07:31.156446 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:31.156453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:31.158797 (+ 2344us) server_negotiation.cc:407] Connection header received
0928 02:07:31.159109 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:31.159116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:31.159208 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:31.159281 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:31.161403 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.162489 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.164499 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.165290 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.165486 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:31.169336 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:31.169341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:31.169344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:31.169379 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:31.172337 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:31.173149 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:31.173153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:31.173155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:31.173296 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:31.173737 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:31.173741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:31.173743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:31.174032 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:31.174302 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:31.174596 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 02:07:31.174776 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:07:31.180158 14661 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:07:31.187947 14661 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:07:31.212375 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.202925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:42201 (local address 127.0.0.1:41318)
0928 02:07:31.204959 (+ 2034us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:31.204972 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:31.205107 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:31.205698 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:31.205705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:31.205732 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:07:31.206055 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:31.206065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.207742 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.207745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:31.209394 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:31.209406 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.211357 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.211364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:31.211563 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:31.211631 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 02:07:31.211772 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 02:07:31.211989 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1943,"thread_start_us":1845,"threads_started":1}
I20260928 02:07:31.213184 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.203919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41318 (local address 127.1.182.130:42201)
0928 02:07:31.204513 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:31.204521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:31.205064 (+ 543us) server_negotiation.cc:407] Connection header received
0928 02:07:31.205263 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:31.205275 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:31.205486 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:31.205695 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:07:31.206220 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.207585 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.209544 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.211136 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.211294 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:31.211537 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:31.212637 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0928 02:07:31.212826 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":305,"threads_started":1}
I20260928 02:07:31.217119 14939 tablet_service.cc:1467] Processing CreateTablet for tablet a943e18c51fe4d128c4c7456494af291 (DEFAULT_TABLE table=test-table [id=663c3a343fd64a4ebf7cac6046601de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:07:31.218752 14939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a943e18c51fe4d128c4c7456494af291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:07:31.221118 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.207328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:33597 (local address 127.0.0.1:56036)
0928 02:07:31.212014 (+ 4686us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:31.212036 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:31.212223 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:31.213157 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:31.213165 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:31.213191 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:07:31.213573 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:31.213582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.215992 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.215997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:31.218071 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:31.218084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.220330 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.220336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:31.220496 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:31.220529 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:07:31.220610 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:07:31.220755 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4569,"thread_start_us":4439,"threads_started":1}
I20260928 02:07:31.221930 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.207849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56036 (local address 127.1.182.131:33597)
0928 02:07:31.208355 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:31.208365 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:31.212123 (+ 3758us) server_negotiation.cc:407] Connection header received
0928 02:07:31.212399 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:31.212411 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:31.212687 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:31.212850 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:07:31.213728 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.215837 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.218263 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.219811 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.219912 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:31.220137 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:31.221434 (+ 1297us) server_negotiation.cc:299] Negotiation successful
0928 02:07:31.221595 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":232,"threads_started":1}
I20260928 02:07:31.224422 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.202861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:38931 (local address 127.0.0.1:54288)
0928 02:07:31.203637 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:31.203681 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:31.203836 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:31.214726 (+ 10890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:31.214731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:31.214756 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:07:31.215025 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:31.215035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.217702 (+ 2667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.217707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:31.220781 (+ 3074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:31.220792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.223660 (+ 2868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.223666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:31.223831 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:31.223869 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:07:31.223935 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:07:31.224063 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":397,"threads_started":1}
I20260928 02:07:31.225668 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.203375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54288 (local address 127.1.182.129:38931)
0928 02:07:31.214133 (+ 10758us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:31.214141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:31.214169 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:07:31.214262 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:31.214269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:31.214503 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:31.215754 (+ 1251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:07:31.215990 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.217548 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.221981 (+ 4433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.223421 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.224918 (+ 1497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:31.225143 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:31.225215 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 02:07:31.225362 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10581,"thread_start_us":19716,"threads_started":1}
I20260928 02:07:31.226052 15076 tablet_service.cc:1467] Processing CreateTablet for tablet a943e18c51fe4d128c4c7456494af291 (DEFAULT_TABLE table=test-table [id=663c3a343fd64a4ebf7cac6046601de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:07:31.227476 15076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a943e18c51fe4d128c4c7456494af291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:07:31.229259 14794 tablet_service.cc:1467] Processing CreateTablet for tablet a943e18c51fe4d128c4c7456494af291 (DEFAULT_TABLE table=test-table [id=663c3a343fd64a4ebf7cac6046601de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:07:31.230649 14794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a943e18c51fe4d128c4c7456494af291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:07:31.246989 15169 tablet_bootstrap.cc:492] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8: Bootstrap starting.
I20260928 02:07:31.249770 15171 tablet_bootstrap.cc:492] T a943e18c51fe4d128c4c7456494af291 P 2055bca407f04439a1fb9b08717157b7: Bootstrap starting.
I20260928 02:07:31.252053 15170 tablet_bootstrap.cc:492] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81: Bootstrap starting.
I20260928 02:07:31.253264 15171 tablet_bootstrap.cc:654] T a943e18c51fe4d128c4c7456494af291 P 2055bca407f04439a1fb9b08717157b7: Neither blocks nor log segments found. Creating new log.
I20260928 02:07:31.254550 15171 log.cc:824] T a943e18c51fe4d128c4c7456494af291 P 2055bca407f04439a1fb9b08717157b7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:07:31.255046 15169 tablet_bootstrap.cc:654] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8: Neither blocks nor log segments found. Creating new log.
I20260928 02:07:31.256224 15170 tablet_bootstrap.cc:654] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81: Neither blocks nor log segments found. Creating new log.
I20260928 02:07:31.256394 15169 log.cc:824] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:07:31.257529 15170 log.cc:824] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81: Log is configured to *not* fsync() on all Append() calls
I20260928 02:07:31.263931 15171 tablet_bootstrap.cc:492] T a943e18c51fe4d128c4c7456494af291 P 2055bca407f04439a1fb9b08717157b7: No bootstrap required, opened a new log
I20260928 02:07:31.264336 15171 ts_tablet_manager.cc:1397] T a943e18c51fe4d128c4c7456494af291 P 2055bca407f04439a1fb9b08717157b7: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20260928 02:07:31.274492 15171 raft_consensus.cc:348] T a943e18c51fe4d128c4c7456494af291 P 2055bca407f04439a1fb9b08717157b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2055bca407f04439a1fb9b08717157b7" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 38931 } } peers { permanent_uuid: "ac84e640a1094fd89c94dfa8146ac3e8" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42201 } } peers { permanent_uuid: "bdac00a0660849e6b5b2437be1385a81" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33597 } }
I20260928 02:07:31.275179 15171 raft_consensus.cc:374] T a943e18c51fe4d128c4c7456494af291 P 2055bca407f04439a1fb9b08717157b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:07:31.275386 15171 raft_consensus.cc:729] T a943e18c51fe4d128c4c7456494af291 P 2055bca407f04439a1fb9b08717157b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2055bca407f04439a1fb9b08717157b7, State: Initialized, Role: FOLLOWER
I20260928 02:07:31.276232 15171 consensus_queue.cc:260] T a943e18c51fe4d128c4c7456494af291 P 2055bca407f04439a1fb9b08717157b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2055bca407f04439a1fb9b08717157b7" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 38931 } } peers { permanent_uuid: "ac84e640a1094fd89c94dfa8146ac3e8" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42201 } } peers { permanent_uuid: "bdac00a0660849e6b5b2437be1385a81" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33597 } }
I20260928 02:07:31.278247 15169 tablet_bootstrap.cc:492] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8: No bootstrap required, opened a new log
I20260928 02:07:31.278561 15169 ts_tablet_manager.cc:1397] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.007s
I20260928 02:07:31.278944 15170 tablet_bootstrap.cc:492] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81: No bootstrap required, opened a new log
I20260928 02:07:31.279256 15170 ts_tablet_manager.cc:1397] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.017s
I20260928 02:07:31.279783 14859 heartbeater.cc:503] Master 127.1.182.190:33515 was elected leader, sending a full tablet report...
I20260928 02:07:31.280400 15171 ts_tablet_manager.cc:1428] T a943e18c51fe4d128c4c7456494af291 P 2055bca407f04439a1fb9b08717157b7: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260928 02:07:31.289285 15169 raft_consensus.cc:348] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2055bca407f04439a1fb9b08717157b7" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 38931 } } peers { permanent_uuid: "ac84e640a1094fd89c94dfa8146ac3e8" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42201 } } peers { permanent_uuid: "bdac00a0660849e6b5b2437be1385a81" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33597 } }
I20260928 02:07:31.289758 15169 raft_consensus.cc:374] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:07:31.289868 15169 raft_consensus.cc:729] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac84e640a1094fd89c94dfa8146ac3e8, State: Initialized, Role: FOLLOWER
I20260928 02:07:31.290190 15170 raft_consensus.cc:348] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2055bca407f04439a1fb9b08717157b7" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 38931 } } peers { permanent_uuid: "ac84e640a1094fd89c94dfa8146ac3e8" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42201 } } peers { permanent_uuid: "bdac00a0660849e6b5b2437be1385a81" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33597 } }
I20260928 02:07:31.290634 15170 raft_consensus.cc:374] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:07:31.290540 15169 consensus_queue.cc:260] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2055bca407f04439a1fb9b08717157b7" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 38931 } } peers { permanent_uuid: "ac84e640a1094fd89c94dfa8146ac3e8" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42201 } } peers { permanent_uuid: "bdac00a0660849e6b5b2437be1385a81" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33597 } }
I20260928 02:07:31.290737 15170 raft_consensus.cc:729] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdac00a0660849e6b5b2437be1385a81, State: Initialized, Role: FOLLOWER
I20260928 02:07:31.291489 15170 consensus_queue.cc:260] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2055bca407f04439a1fb9b08717157b7" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 38931 } } peers { permanent_uuid: "ac84e640a1094fd89c94dfa8146ac3e8" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42201 } } peers { permanent_uuid: "bdac00a0660849e6b5b2437be1385a81" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33597 } }
I20260928 02:07:31.292552 15169 ts_tablet_manager.cc:1428] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260928 02:07:31.293598 15141 heartbeater.cc:503] Master 127.1.182.190:33515 was elected leader, sending a full tablet report...
I20260928 02:07:31.294225 15170 ts_tablet_manager.cc:1428] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260928 02:07:31.333652 15146 tablet.cc:2369] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:07:31.338816 15006 tablet.cc:2369] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:07:31.418722 14860 tablet.cc:2369] T a943e18c51fe4d128c4c7456494af291 P 2055bca407f04439a1fb9b08717157b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:07:31.465006 15177 raft_consensus.cc:482] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:31.465407 15177 raft_consensus.cc:504] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2055bca407f04439a1fb9b08717157b7" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 38931 } } peers { permanent_uuid: "ac84e640a1094fd89c94dfa8146ac3e8" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42201 } } peers { permanent_uuid: "bdac00a0660849e6b5b2437be1385a81" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33597 } }
I20260928 02:07:31.467145 15177 leader_election.cc:288] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2055bca407f04439a1fb9b08717157b7 (127.1.182.129:38931), ac84e640a1094fd89c94dfa8146ac3e8 (127.1.182.130:42201)
I20260928 02:07:31.473459 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.467572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:38931 (local address 127.1.182.131:51905)
0928 02:07:31.467863 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:31.467883 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:31.468009 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:31.468571 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:31.468575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:31.468605 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:07:31.468800 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:31.468830 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.470204 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.470209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:31.471710 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:31.471721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.472765 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.472771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:31.472946 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:31.472969 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:07:31.473069 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 02:07:31.473194 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":25}
I20260928 02:07:31.473459 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.467955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:51905 (local address 127.1.182.129:38931)
0928 02:07:31.468226 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:31.468233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:31.468259 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:07:31.468345 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:31.468350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:31.468427 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:31.468552 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:07:31.468970 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.470067 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.471853 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.472610 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.472772 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:31.472846 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:31.473095 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 02:07:31.473221 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:07:31.474499 14814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a943e18c51fe4d128c4c7456494af291" candidate_uuid: "bdac00a0660849e6b5b2437be1385a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2055bca407f04439a1fb9b08717157b7" is_pre_election: true
I20260928 02:07:31.474820 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.467569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:47531 (local address 127.1.182.130:42201)
0928 02:07:31.467795 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:31.467802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:31.468650 (+ 848us) server_negotiation.cc:407] Connection header received
0928 02:07:31.468908 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:31.468915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:31.468989 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:31.469082 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:07:31.469573 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.470313 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.472782 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.473943 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.474076 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:31.474140 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:31.474466 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 02:07:31.474589 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:07:31.474921 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.467812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:42201 (local address 127.1.182.131:47531)
0928 02:07:31.468553 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:31.468570 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:31.468753 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:31.469093 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:31.469097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:31.469119 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:07:31.469436 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:31.469444 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.470463 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.470468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:31.472608 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:31.472621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.474109 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.474116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:31.474295 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:31.474330 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:07:31.474429 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 02:07:31.474556 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":26,"thread_start_us":451,"threads_started":1}
I20260928 02:07:31.475142 14814 raft_consensus.cc:2454] T a943e18c51fe4d128c4c7456494af291 P 2055bca407f04439a1fb9b08717157b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdac00a0660849e6b5b2437be1385a81 in term 0.
I20260928 02:07:31.475919 15029 leader_election.cc:302] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2055bca407f04439a1fb9b08717157b7, bdac00a0660849e6b5b2437be1385a81; no voters:
I20260928 02:07:31.476047 14959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a943e18c51fe4d128c4c7456494af291" candidate_uuid: "bdac00a0660849e6b5b2437be1385a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac84e640a1094fd89c94dfa8146ac3e8" is_pre_election: true
I20260928 02:07:31.476315 15177 raft_consensus.cc:2781] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:07:31.476464 15177 raft_consensus.cc:482] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:07:31.476583 15177 raft_consensus.cc:3037] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:07:31.476563 14959 raft_consensus.cc:2454] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdac00a0660849e6b5b2437be1385a81 in term 0.
I20260928 02:07:31.478957 15177 raft_consensus.cc:504] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2055bca407f04439a1fb9b08717157b7" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 38931 } } peers { permanent_uuid: "ac84e640a1094fd89c94dfa8146ac3e8" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42201 } } peers { permanent_uuid: "bdac00a0660849e6b5b2437be1385a81" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33597 } }
I20260928 02:07:31.479553 15177 leader_election.cc:288] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 [CANDIDATE]: Term 1 election: Requested vote from peers 2055bca407f04439a1fb9b08717157b7 (127.1.182.129:38931), ac84e640a1094fd89c94dfa8146ac3e8 (127.1.182.130:42201)
I20260928 02:07:31.479956 14814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a943e18c51fe4d128c4c7456494af291" candidate_uuid: "bdac00a0660849e6b5b2437be1385a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2055bca407f04439a1fb9b08717157b7"
I20260928 02:07:31.480036 14959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a943e18c51fe4d128c4c7456494af291" candidate_uuid: "bdac00a0660849e6b5b2437be1385a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac84e640a1094fd89c94dfa8146ac3e8"
I20260928 02:07:31.480136 14814 raft_consensus.cc:3037] T a943e18c51fe4d128c4c7456494af291 P 2055bca407f04439a1fb9b08717157b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:07:31.480232 14959 raft_consensus.cc:3037] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:07:31.482271 14814 raft_consensus.cc:2454] T a943e18c51fe4d128c4c7456494af291 P 2055bca407f04439a1fb9b08717157b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdac00a0660849e6b5b2437be1385a81 in term 1.
I20260928 02:07:31.482637 15029 leader_election.cc:302] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2055bca407f04439a1fb9b08717157b7, bdac00a0660849e6b5b2437be1385a81; no voters:
I20260928 02:07:31.482887 15177 raft_consensus.cc:2781] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:07:31.483213 14959 raft_consensus.cc:2454] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdac00a0660849e6b5b2437be1385a81 in term 1.
I20260928 02:07:31.483454 15177 raft_consensus.cc:686] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 [term 1 LEADER]: Becoming Leader. State: Replica: bdac00a0660849e6b5b2437be1385a81, State: Running, Role: LEADER
I20260928 02:07:31.484273 15177 consensus_queue.cc:237] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2055bca407f04439a1fb9b08717157b7" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 38931 } } peers { permanent_uuid: "ac84e640a1094fd89c94dfa8146ac3e8" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42201 } } peers { permanent_uuid: "bdac00a0660849e6b5b2437be1385a81" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33597 } }
I20260928 02:07:31.488864 14661 catalog_manager.cc:5495] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdac00a0660849e6b5b2437be1385a81 (127.1.182.131). New cstate: current_term: 1 leader_uuid: "bdac00a0660849e6b5b2437be1385a81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2055bca407f04439a1fb9b08717157b7" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 38931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac84e640a1094fd89c94dfa8146ac3e8" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdac00a0660849e6b5b2437be1385a81" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33597 } health_report { overall_health: HEALTHY } } }
I20260928 02:07:31.511061 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.504941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56038 (local address 127.1.182.131:33597)
0928 02:07:31.505154 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:31.505160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:31.505275 (+ 115us) server_negotiation.cc:407] Connection header received
0928 02:07:31.505454 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:31.505459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:31.505568 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:31.505752 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:07:31.506061 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.507200 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.508644 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.509256 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.509377 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:31.509837 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:07:31.510522 (+ 685us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:07:31.510670 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:31.510714 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 02:07:31.510790 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:07:31.519223 14814 raft_consensus.cc:1261] T a943e18c51fe4d128c4c7456494af291 P 2055bca407f04439a1fb9b08717157b7 [term 1 FOLLOWER]: Refusing update from remote peer bdac00a0660849e6b5b2437be1385a81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:07:31.519223 14959 raft_consensus.cc:1261] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8 [term 1 FOLLOWER]: Refusing update from remote peer bdac00a0660849e6b5b2437be1385a81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:07:31.520231 15177 consensus_queue.cc:1035] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac84e640a1094fd89c94dfa8146ac3e8" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:07:31.520502 15179 consensus_queue.cc:1035] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2055bca407f04439a1fb9b08717157b7" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 38931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:07:31.541895 15182 mvcc.cc:205] Tried to move back new op lower bound from 7334138886211944448 to 7334138886083194880. Current Snapshot: MvccSnapshot[applied={T|T < 7334138886211944448}]
I20260928 02:07:31.558018 14661 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:07:31.558319 14661 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:07:31.562568 14661 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2eba3cad751f4dcba8a14a925e7859fe: Sending DeleteTablet for 3 replicas of tablet a943e18c51fe4d128c4c7456494af291
I20260928 02:07:31.564138 14794 tablet_service.cc:1514] Processing DeleteTablet for tablet a943e18c51fe4d128c4c7456494af291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:07:31 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:54288
I20260928 02:07:31.564378 14939 tablet_service.cc:1514] Processing DeleteTablet for tablet a943e18c51fe4d128c4c7456494af291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:07:31 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:41318
I20260928 02:07:31.565070 15076 tablet_service.cc:1514] Processing DeleteTablet for tablet a943e18c51fe4d128c4c7456494af291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:07:31 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:56036
I20260928 02:07:31.576462 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 14698
I20260928 02:07:31.597066 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.578252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54290 (local address 127.1.182.129:38931)
0928 02:07:31.578470 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:31.578479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:31.578502 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:07:31.578589 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:31.578597 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:31.578699 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:31.578813 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:31.581139 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.582536 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.584392 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.585308 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.585431 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:31.588762 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:31.588785 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:31.588809 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:31.588860 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:31.593373 (+ 4513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:31.594163 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:31.594176 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:31.594182 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:31.594366 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:31.594752 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:31.594760 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:31.594766 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:31.595348 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:31.595667 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:31.595912 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 02:07:31.596610 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:07:31.598188 14854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:07:31.606164 15194 tablet_replica.cc:330] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81: stopping tablet replica
I20260928 02:07:31.607548 15193 tablet_replica.cc:330] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8: stopping tablet replica
I20260928 02:07:31.608035 15193 raft_consensus.cc:2229] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:07:31.608592 15193 raft_consensus.cc:2258] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:07:31.609264 15194 raft_consensus.cc:2229] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:07:31.609985 15194 raft_consensus.cc:2258] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:07:31.612067 15193 ts_tablet_manager.cc:1905] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:07:31.612174 15194 ts_tablet_manager.cc:1905] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:07:31.619442 15194 ts_tablet_manager.cc:1918] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:07:31.619745 15194 log.cc:1196] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/a943e18c51fe4d128c4c7456494af291
I20260928 02:07:31.620390 15194 ts_tablet_manager.cc:1939] T a943e18c51fe4d128c4c7456494af291 P bdac00a0660849e6b5b2437be1385a81: Deleting consensus metadata
I20260928 02:07:31.622295 14647 catalog_manager.cc:4841] TS bdac00a0660849e6b5b2437be1385a81 (127.1.182.131:33597): tablet a943e18c51fe4d128c4c7456494af291 (table test-table [id=663c3a343fd64a4ebf7cac6046601de5]) successfully deleted
I20260928 02:07:31.628795 15193 ts_tablet_manager.cc:1918] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:07:31.629093 15193 log.cc:1196] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/a943e18c51fe4d128c4c7456494af291
I20260928 02:07:31.629711 15193 ts_tablet_manager.cc:1939] T a943e18c51fe4d128c4c7456494af291 P ac84e640a1094fd89c94dfa8146ac3e8: Deleting consensus metadata
I20260928 02:07:31.632254 14647 catalog_manager.cc:4841] TS ac84e640a1094fd89c94dfa8146ac3e8 (127.1.182.130:42201): tablet a943e18c51fe4d128c4c7456494af291 (table test-table [id=663c3a343fd64a4ebf7cac6046601de5]) successfully deleted
I20260928 02:07:31.763882 15192 tablet_replica.cc:330] T a943e18c51fe4d128c4c7456494af291 P 2055bca407f04439a1fb9b08717157b7: stopping tablet replica
I20260928 02:07:31.764367 15192 raft_consensus.cc:2229] T a943e18c51fe4d128c4c7456494af291 P 2055bca407f04439a1fb9b08717157b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:07:31.764861 15192 raft_consensus.cc:2258] T a943e18c51fe4d128c4c7456494af291 P 2055bca407f04439a1fb9b08717157b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:07:31.765604 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 14698
W20260928 02:07:31.789222 14648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.182.129:38931 (error 108)
I20260928 02:07:31.789896 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 14861
I20260928 02:07:31.792354 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.791604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:38931 (local address 127.0.0.1:54306)
0928 02:07:31.792018 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:31.792149 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:38931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"threads_started":1}
W20260928 02:07:31.792865 14648 catalog_manager.cc:4584] TS 2055bca407f04439a1fb9b08717157b7 (127.1.182.129:38931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a943e18c51fe4d128c4c7456494af291: Network error: Client connection negotiation failed: client connection to 127.1.182.129:38931: connect: Connection refused (error 111)
I20260928 02:07:31.813133 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.790890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41328 (local address 127.1.182.130:42201)
0928 02:07:31.791059 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:31.791087 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:31.791118 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:07:31.791190 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:31.791195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:31.791278 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:31.791375 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:31.793435 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.794548 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.796547 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.797186 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.797316 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:31.806238 (+ 8922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:31.806257 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:31.806278 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:31.806326 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:31.810094 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:31.810887 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:31.810896 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:31.810900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:31.811102 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:31.811499 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:31.811504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:31.811508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:31.811946 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:31.812235 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:31.812534 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 02:07:31.812746 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:07:31.813783 15000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:07:31.846740 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.846161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:38931 (local address 127.0.0.1:54316)
0928 02:07:31.846450 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:31.846561 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:38931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260928 02:07:31.848276 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.847729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:38931 (local address 127.0.0.1:54320)
0928 02:07:31.847999 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:31.848075 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:38931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"threads_started":1}
I20260928 02:07:31.920861 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 14861
I20260928 02:07:31.924194 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.920627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:38931 (local address 127.0.0.1:54334)
0928 02:07:31.923880 (+ 3253us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:31.924030 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:38931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3130}
I20260928 02:07:31.925196 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.924878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:38931 (local address 127.0.0.1:54348)
0928 02:07:31.925019 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:31.925099 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:38931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 02:07:31.944286 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 15009
I20260928 02:07:31.960816 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:31.945148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56044 (local address 127.1.182.131:33597)
0928 02:07:31.945367 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:31.945373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:31.945395 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:07:31.945475 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:31.945482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:31.945571 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:31.945702 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:31.946893 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.948146 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.949639 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:31.950311 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:31.950454 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:31.953118 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:31.953150 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:31.953196 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:31.953249 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:31.957526 (+ 4277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:31.958215 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:31.958230 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:31.958234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:31.958420 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:31.958793 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:31.958798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:31.958801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:31.959628 (+ 827us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:31.959974 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:31.960154 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 02:07:31.960392 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 02:07:31.961570 15136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:07:32.005880 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:32.005403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:38931 (local address 127.0.0.1:54354)
0928 02:07:32.005626 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:32.005717 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:38931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 02:07:32.006937 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:32.006589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:38931 (local address 127.0.0.1:54368)
0928 02:07:32.006765 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:32.006829 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:38931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 02:07:32.089022 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 15009
I20260928 02:07:32.105378 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 14627
I20260928 02:07:32.106524 14691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:07:32.216926 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 14627
I20260928 02:07:32.238063 1754 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 02:07:32.572701 1754 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 02:07:32.572937 1754 mini_ranger.cc:66] Stopping Ranger...
I20260928 02:07:32.918701 1754 mini_ranger.cc:68] Stopped Ranger
2026-09-28 02:07:32.918 UTC [13673] LOG: received smart shutdown request
2026-09-28 02:07:32.922 UTC [13673] LOG: background worker "logical replication launcher" (PID 13682) exited with exit code 1
2026-09-28 02:07:32.922 UTC [13677] LOG: shutting down
2026-09-28 02:07:33.090 UTC [13673] LOG: database system is shut down
2026-09-28T02:07:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52670 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:07:33 dist-test-slave-23bp krb5kdc[15210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:07:33 dist-test-slave-23bp krb5kdc[15210](info): set up 2 sockets
Sep 28 02:07:33 dist-test-slave-23bp krb5kdc[15210](info): commencing operation
krb5kdc: starting...
W20260928 02:07:34.377815 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:07:34 dist-test-slave-23bp krb5kdc[15210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:07:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:07:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:07:34.606832 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:37209
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:37885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:37209,127.1.182.189:38295,127.1.182.188:46655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:07:34.932420 15223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:34.932782 15223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:34.932940 15223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:34.940661 15223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:07:34.940811 15223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:34.940922 15223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:34.941033 15223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:07:34.941123 15223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:07:34.951191 15223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:37885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:37209,127.1.182.189:38295,127.1.182.188:46655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:37209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:34.952610 15223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:34.953927 15223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:34.966254 15228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:34.966344 15229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:34.966285 15231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:34.967941 15223 server_base.cc:1034] running on GCE node
I20260928 02:07:34.968858 15223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:34.970435 15223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:34.971698 15223 hybrid_clock.cc:648] HybridClock initialized: now 1790561254971655 us; error 60 us; skew 500 ppm
I20260928 02:07:34.972084 15223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:34 dist-test-slave-23bp krb5kdc[15210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561254, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:34.981865 15223 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:07:34.983763 15223 webserver.cc:460] Webserver started at http://127.1.182.190:37833/ using document root <none> and password file <none>
I20260928 02:07:34.984434 15223 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:34.984532 15223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:07:34.984778 15223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:07:34.987756 15223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c"
format_stamp: "Formatted at 2026-09-28 02:07:34 on dist-test-slave-23bp"
server_key: "c6064287bda6ebed2afa803e2740e110"
server_key_iv: "fb8609786bd9776d2e1fa5d3752e395d"
server_key_version: "encryptionkey@0"
I20260928 02:07:34.988498 15223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c"
format_stamp: "Formatted at 2026-09-28 02:07:34 on dist-test-slave-23bp"
server_key: "c6064287bda6ebed2afa803e2740e110"
server_key_iv: "fb8609786bd9776d2e1fa5d3752e395d"
server_key_version: "encryptionkey@0"
I20260928 02:07:34.993561 15223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 02:07:34.997002 15238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:34.998495 15223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:07:34.998672 15223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c"
format_stamp: "Formatted at 2026-09-28 02:07:34 on dist-test-slave-23bp"
server_key: "c6064287bda6ebed2afa803e2740e110"
server_key_iv: "fb8609786bd9776d2e1fa5d3752e395d"
server_key_version: "encryptionkey@0"
I20260928 02:07:34.998869 15223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:35.015450 15223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:35.019512 15223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:35.019804 15223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:35.037912 15223 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:37209
I20260928 02:07:35.037951 15289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:37209 every 8 connection(s)
I20260928 02:07:35.039271 15223 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:07:35.042898 15290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:07:35.048848 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 15223
I20260928 02:07:35.049075 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:07:35.049700 1754 external_mini_cluster.cc:1428] Setting key ec2c68ad978cc1c700d0aa140d6acb3a
I20260928 02:07:35.049335 15290 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:07:35 dist-test-slave-23bp krb5kdc[15210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561254, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.182.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.189@KRBTEST.COM" created.
I20260928 02:07:35.075479 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.052465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:37209 (local address 127.0.0.1:54640)
0928 02:07:35.057258 (+ 4793us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.057320 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:35.059303 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:35.060099 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:35.060105 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:35.060809 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:35.061205 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:35.061222 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:35.062876 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:35.062885 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:35.063981 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:35.063993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:35.064865 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:35.064872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:35.065051 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:35.065715 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:35.065736 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:35.068287 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:35.071514 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:35.071526 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:35.071538 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:35.071915 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:35.072325 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:35.072329 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:35.072333 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:35.072477 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:35.073191 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:35.073200 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:35.073415 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 02:07:35.073655 (+ 240us) client_negotiation.cc:240] Negotiation successful
0928 02:07:35.074346 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4413,"mutex_wait_us":230,"thread_start_us":4148,"threads_started":1}
I20260928 02:07:35.075526 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.052465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54640 (local address 127.1.182.190:37209)
0928 02:07:35.057970 (+ 5505us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:35.057980 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:35.058035 (+ 55us) server_negotiation.cc:407] Connection header received
0928 02:07:35.059661 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:35.059666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:35.059885 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:35.060092 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:35.061393 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:35.062727 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:35.064123 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:35.064647 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:35.064835 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:35.068505 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:35.068512 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:35.068523 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:35.068553 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:35.071397 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:35.072041 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:35.072054 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:35.072056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:35.072214 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:35.072583 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:35.072588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:35.072595 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:35.073052 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:35.073264 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:35.073780 (+ 516us) server_negotiation.cc:299] Negotiation successful
0928 02:07:35.074213 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":5260,"spinlock_wait_cycles":29056,"thread_start_us":5043,"threads_started":1}
I20260928 02:07:35.078325 15290 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:07:35.079454 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.079016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38295 (local address 127.0.0.1:58674)
0928 02:07:35.079238 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.079345 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260928 02:07:35.079708 15242 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38295: connect: Connection refused (error 111)
I20260928 02:07:35.081391 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.080954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38295 (local address 127.0.0.1:58690)
0928 02:07:35.081134 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.081226 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 02:07:35.081640 15290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.189:38295: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38295: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.189.keytab.
Entry for principal kudu/127.1.182.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.189.keytab.
I20260928 02:07:35.102612 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.189:38295
--webserver_interface=127.1.182.189
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.189.keytab
--principal=kudu/127.1.182.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:37885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:37209,127.1.182.189:38295,127.1.182.188:46655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:07:35.130999 15290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } attempt: 1
I20260928 02:07:35.132254 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.131686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38295 (local address 127.0.0.1:58696)
0928 02:07:35.131929 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.132093 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 02:07:35.133239 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.132866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38295 (local address 127.0.0.1:58700)
0928 02:07:35.133010 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.133094 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 02:07:35.133432 15290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.189:38295: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38295: connect: Connection refused (error 111)
I20260928 02:07:35.201696 15290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } attempt: 2
I20260928 02:07:35.203188 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.202446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38295 (local address 127.0.0.1:58706)
0928 02:07:35.202859 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.202977 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226}
I20260928 02:07:35.204401 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.204046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38295 (local address 127.0.0.1:58710)
0928 02:07:35.204176 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.204274 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 02:07:35.204689 15290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.189:38295: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38295: connect: Connection refused (error 111)
I20260928 02:07:35.295950 15290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } attempt: 3
I20260928 02:07:35.297153 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.296730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38295 (local address 127.0.0.1:58716)
0928 02:07:35.296920 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.297025 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 02:07:35.298233 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.297902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38295 (local address 127.0.0.1:58724)
0928 02:07:35.298025 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.298087 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 02:07:35.298493 15290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.189:38295: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38295: connect: Connection refused (error 111)
W20260928 02:07:35.429975 15297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:35.430264 15297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:35.430423 15297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:35.437520 15297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:07:35.437624 15297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:35.437718 15297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:35.437819 15297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:07:35.437893 15297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:07:35.441740 15290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } attempt: 4
I20260928 02:07:35.442953 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.442424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38295 (local address 127.0.0.1:58736)
0928 02:07:35.442732 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.442819 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260928 02:07:35.443965 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.443625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38295 (local address 127.0.0.1:58742)
0928 02:07:35.443764 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.443851 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 02:07:35.444177 15290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.189:38295: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38295: connect: Connection refused (error 111)
I20260928 02:07:35.448171 15297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:37885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:37209,127.1.182.189:38295,127.1.182.188:46655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.189.keytab
--principal=kudu/127.1.182.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.189:38295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.182.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:35.449462 15297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:35.450776 15297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:35.463008 15305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:35.463032 15302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:35.463421 15303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:35.468690 15297 server_base.cc:1034] running on GCE node
I20260928 02:07:35.469461 15297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:35.470862 15297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:35.472105 15297 hybrid_clock.cc:648] HybridClock initialized: now 1790561255472032 us; error 66 us; skew 500 ppm
I20260928 02:07:35.472447 15297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:35 dist-test-slave-23bp krb5kdc[15210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561255, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:35.475939 15297 init.cc:377] Logged in from keytab as kudu/127.1.182.189@KRBTEST.COM (short username kudu)
I20260928 02:07:35.477433 15297 webserver.cc:460] Webserver started at http://127.1.182.189:43469/ using document root <none> and password file <none>
I20260928 02:07:35.478137 15297 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:35.478274 15297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:07:35.478564 15297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:07:35.481515 15297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-1/data/instance:
uuid: "69610fca01d64f868a63b7a228533e7f"
format_stamp: "Formatted at 2026-09-28 02:07:35 on dist-test-slave-23bp"
server_key: "47342eddca9a7287ad81f9318cc66051"
server_key_iv: "426f2049c768782f839e13cfaadecc99"
server_key_version: "encryptionkey@0"
I20260928 02:07:35.482290 15297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-1/wal/instance:
uuid: "69610fca01d64f868a63b7a228533e7f"
format_stamp: "Formatted at 2026-09-28 02:07:35 on dist-test-slave-23bp"
server_key: "47342eddca9a7287ad81f9318cc66051"
server_key_iv: "426f2049c768782f839e13cfaadecc99"
server_key_version: "encryptionkey@0"
I20260928 02:07:35.487365 15297 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 02:07:35.490690 15312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:35.492008 15297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:07:35.492200 15297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-1/wal
uuid: "69610fca01d64f868a63b7a228533e7f"
format_stamp: "Formatted at 2026-09-28 02:07:35 on dist-test-slave-23bp"
server_key: "47342eddca9a7287ad81f9318cc66051"
server_key_iv: "426f2049c768782f839e13cfaadecc99"
server_key_version: "encryptionkey@0"
I20260928 02:07:35.492388 15297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:35.520454 15297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:35.524842 15297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:35.525157 15297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:35.543612 15297 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.189:38295
I20260928 02:07:35.543660 15363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.189:38295 every 8 connection(s)
I20260928 02:07:35.545040 15297 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-1/data/info.pb
I20260928 02:07:35.548664 15364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:07:35.554374 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 15297
I20260928 02:07:35.554562 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-1/wal/instance
I20260928 02:07:35.555018 1754 external_mini_cluster.cc:1428] Setting key 6d1e04f7e0b058ad87abd31ba6ec4a7b
I20260928 02:07:35.555222 15364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:07:35 dist-test-slave-23bp krb5kdc[15210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561255, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.189@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:07:35.579034 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.557738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.1.182.190:37209)
0928 02:07:35.557905 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:35.557909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:35.559397 (+ 1488us) server_negotiation.cc:407] Connection header received
0928 02:07:35.561544 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:35.561553 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:35.561649 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:35.561713 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:35.563725 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:35.564890 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:35.566521 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:35.567337 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:35.567503 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:35.570949 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:35.570958 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:35.570965 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:35.571013 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:35.575187 (+ 4174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:35.576411 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:35.576416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:35.576419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:35.576553 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:35.577127 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:35.577132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:35.577134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:35.577431 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:35.577555 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:35.578629 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0928 02:07:35.578807 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.1.182.188@KRBTEST.COM; defaulting to no policy
I20260928 02:07:35.580426 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.558358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:37209 (local address 127.0.0.1:54644)
0928 02:07:35.559233 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.559278 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:35.561247 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:35.562170 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:35.562182 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:35.562821 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:35.563517 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:35.563536 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:35.565041 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:35.565044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:35.566290 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:35.566301 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:35.567566 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:35.567571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:35.567761 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:35.568454 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:35.568479 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:35.570710 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:35.575367 (+ 4657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:35.575380 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:35.575405 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:35.576237 (+ 832us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:35.576676 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:35.576684 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:35.576689 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:35.576960 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:35.577582 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:35.577593 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:35.578125 (+ 532us) client_negotiation.cc:769] Sending connection context
0928 02:07:35.578541 (+ 416us) client_negotiation.cc:240] Negotiation successful
0928 02:07:35.579177 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":362,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.188@KRBTEST.COM" created.
I20260928 02:07:35.582399 15364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:07:35 dist-test-slave-23bp krb5kdc[15210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561255, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.189@KRBTEST.COM for kudu/127.1.182.189@KRBTEST.COM
I20260928 02:07:35.598759 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.583312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38295 (local address 127.0.0.1:58744)
0928 02:07:35.583535 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.583574 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:35.583742 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:35.584553 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:35.584556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:35.584921 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:35.585062 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:35.585069 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:35.587129 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:35.587135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:35.587962 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:35.587973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:35.589184 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:35.589188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:35.589355 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:35.589927 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:35.589944 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:35.592001 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:35.596357 (+ 4356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:35.596361 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:35.596363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:35.596838 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:35.597236 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:35.597240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:35.597242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:35.597380 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:35.598101 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:35.598104 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:35.598215 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 02:07:35.598313 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 02:07:35.598503 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":2}
I20260928 02:07:35.599292 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.583410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58744 (local address 127.1.182.189:38295)
0928 02:07:35.584034 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:35.584044 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:35.584079 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:07:35.584140 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:35.584151 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:35.584397 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:35.584571 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:35.585217 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:35.586956 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:35.588095 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:35.589031 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:35.589186 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:35.592211 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:35.592226 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:35.592246 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:35.592285 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:35.596240 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:35.596976 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:35.596988 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:35.596995 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:35.597135 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:35.597486 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:35.597494 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:35.597500 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:35.597970 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:35.598185 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:35.598505 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 02:07:35.598787 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":427,"thread_start_us":265,"threads_started":1}
I20260928 02:07:35.601743 15364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:07:35.602877 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.602409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:46655 (local address 127.0.0.1:38946)
0928 02:07:35.602597 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.602714 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 02:07:35.603238 15313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
I20260928 02:07:35.604818 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.604525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:46655 (local address 127.0.0.1:38952)
0928 02:07:35.604657 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.604730 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 02:07:35.605221 15364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:46655: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.188.keytab.
Entry for principal kudu/127.1.182.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.188.keytab.
I20260928 02:07:35.614743 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.188:46655
--webserver_interface=127.1.182.188
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.188.keytab
--principal=kudu/127.1.182.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:37885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:37209,127.1.182.189:38295,127.1.182.188:46655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:07:35.654577 15364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } attempt: 1
I20260928 02:07:35.655838 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.655369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:46655 (local address 127.0.0.1:38960)
0928 02:07:35.655587 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.655689 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 02:07:35.656827 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.656469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:46655 (local address 127.0.0.1:38976)
0928 02:07:35.656644 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.656711 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 02:07:35.657111 15364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:46655: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
I20260928 02:07:35.725356 15364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } attempt: 2
I20260928 02:07:35.726492 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.726048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:46655 (local address 127.0.0.1:38986)
0928 02:07:35.726246 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.726343 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 02:07:35.727563 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.727242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:46655 (local address 127.0.0.1:38998)
0928 02:07:35.727378 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.727453 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 02:07:35.727778 15364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:46655: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
I20260928 02:07:35.743422 15290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } attempt: 5
Sep 28 02:07:35 dist-test-slave-23bp krb5kdc[15210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561254, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for kudu/127.1.182.189@KRBTEST.COM
I20260928 02:07:35.757611 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.744223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58748 (local address 127.1.182.189:38295)
0928 02:07:35.744382 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:35.744386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:35.744406 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:07:35.744609 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:35.744612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:35.744690 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:35.744768 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:35.745832 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:35.746527 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:35.748306 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:35.748725 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:35.748821 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:35.752528 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:35.752533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:35.752536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:35.752567 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:35.755224 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:35.755942 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:35.755945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:35.755948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:35.756053 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:35.756427 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:35.756430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:35.756432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:35.756692 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:35.756799 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:35.757221 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 02:07:35.757380 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:07:35.757772 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.744054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38295 (local address 127.0.0.1:58748)
0928 02:07:35.744289 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.744310 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:35.744442 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:35.744842 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:35.744846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:35.745427 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:35.745693 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:35.745706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:35.746666 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:35.746673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:35.748138 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:35.748157 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:35.748875 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:35.748881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:35.749050 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:35.749701 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:35.749723 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:35.752343 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:35.755343 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:35.755348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:35.755352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:35.755840 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:35.756156 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:35.756161 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:35.756165 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:35.756322 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:35.756802 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:35.756807 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:35.756966 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 02:07:35.757125 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 02:07:35.757424 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 02:07:35.758841 15290 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:07:35.759873 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.759513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:46655 (local address 127.0.0.1:39014)
0928 02:07:35.759676 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.759747 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 02:07:35.760787 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.760465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:46655 (local address 127.0.0.1:39028)
0928 02:07:35.760615 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.760691 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 02:07:35.761032 15290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:46655: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
I20260928 02:07:35.812286 15290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } attempt: 1
I20260928 02:07:35.813473 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.813002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:46655 (local address 127.0.0.1:39030)
0928 02:07:35.813192 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.813299 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 02:07:35.814447 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.814125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:46655 (local address 127.0.0.1:39044)
0928 02:07:35.814277 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.814346 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 02:07:35.814651 15290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:46655: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
I20260928 02:07:35.819001 15364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } attempt: 3
I20260928 02:07:35.819947 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.819548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:46655 (local address 127.0.0.1:39054)
0928 02:07:35.819731 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.819811 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 02:07:35.820878 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.820497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:46655 (local address 127.0.0.1:39058)
0928 02:07:35.820706 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.820780 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260928 02:07:35.821077 15364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:46655: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
I20260928 02:07:35.882874 15290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } attempt: 2
I20260928 02:07:35.884058 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.883575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:46655 (local address 127.0.0.1:39072)
0928 02:07:35.883811 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.883896 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 02:07:35.885052 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.884739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:46655 (local address 127.0.0.1:39088)
0928 02:07:35.884886 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.884951 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 02:07:35.885264 15290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:46655: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
W20260928 02:07:35.941941 15371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:35.942313 15371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:35.942471 15371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:35.949817 15371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:07:35.949966 15371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:35.950084 15371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:35.950181 15371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:07:35.950260 15371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:07:35.960361 15371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:37885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:37209,127.1.182.189:38295,127.1.182.188:46655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.188.keytab
--principal=kudu/127.1.182.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.188:46655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.182.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:35.961930 15371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:35.963932 15371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:07:35.964257 15364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } attempt: 4
I20260928 02:07:35.965478 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.964943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:46655 (local address 127.0.0.1:39090)
0928 02:07:35.965189 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.965293 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 02:07:35.966828 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.966457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:46655 (local address 127.0.0.1:39096)
0928 02:07:35.966613 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.966699 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 02:07:35.967101 15364 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:46655: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
W20260928 02:07:35.976758 15376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:35.977813 15379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:35.977859 15377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:35.979211 15371 server_base.cc:1034] running on GCE node
I20260928 02:07:35.979935 15371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:35.981315 15371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:35.982511 15371 hybrid_clock.cc:648] HybridClock initialized: now 1790561255982472 us; error 33 us; skew 500 ppm
I20260928 02:07:35.982815 15371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:35 dist-test-slave-23bp krb5kdc[15210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561255, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:35.985983 15371 init.cc:377] Logged in from keytab as kudu/127.1.182.188@KRBTEST.COM (short username kudu)
I20260928 02:07:35.987564 15371 webserver.cc:460] Webserver started at http://127.1.182.188:43215/ using document root <none> and password file <none>
I20260928 02:07:35.988281 15371 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:35.988409 15371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:07:35.988710 15371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:07:35.991458 15290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } attempt: 3
I20260928 02:07:35.991928 15371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-2/data/instance:
uuid: "a3c5f5f3676a45268cdfba8b788fdc45"
format_stamp: "Formatted at 2026-09-28 02:07:35 on dist-test-slave-23bp"
server_key: "e0d6942706fd965d79fccb3cf3b4a699"
server_key_iv: "f8ae1f80eb3ff9775c725ba8900e79aa"
server_key_version: "encryptionkey@0"
I20260928 02:07:35.992677 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.992146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:46655 (local address 127.0.0.1:39112)
0928 02:07:35.992362 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.992474 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 02:07:35.992904 15371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-2/wal/instance:
uuid: "a3c5f5f3676a45268cdfba8b788fdc45"
format_stamp: "Formatted at 2026-09-28 02:07:35 on dist-test-slave-23bp"
server_key: "e0d6942706fd965d79fccb3cf3b4a699"
server_key_iv: "f8ae1f80eb3ff9775c725ba8900e79aa"
server_key_version: "encryptionkey@0"
I20260928 02:07:35.993791 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:35.993376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:46655 (local address 127.0.0.1:39122)
0928 02:07:35.993580 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:35.993666 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260928 02:07:35.994055 15290 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:46655: Network error: Client connection negotiation failed: client connection to 127.1.182.188:46655: connect: Connection refused (error 111)
I20260928 02:07:35.998610 15371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:07:36.002141 15386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:36.003836 15371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 02:07:36.004021 15371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-2/wal
uuid: "a3c5f5f3676a45268cdfba8b788fdc45"
format_stamp: "Formatted at 2026-09-28 02:07:35 on dist-test-slave-23bp"
server_key: "e0d6942706fd965d79fccb3cf3b4a699"
server_key_iv: "f8ae1f80eb3ff9775c725ba8900e79aa"
server_key_version: "encryptionkey@0"
I20260928 02:07:36.004223 15371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:36.022632 15371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:36.026473 15371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:36.026782 15371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:36.045835 15371 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.188:46655
I20260928 02:07:36.045915 15437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.188:46655 every 8 connection(s)
I20260928 02:07:36.047432 15371 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-2/data/info.pb
I20260928 02:07:36.051254 15438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:07:36.056408 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 15371
I20260928 02:07:36.056659 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561052730740-1754-0/minicluster-data/master-2/wal/instance
I20260928 02:07:36.057356 1754 external_mini_cluster.cc:1428] Setting key cafcbe0d2cd7bc7753d6e116d99e8cb3
I20260928 02:07:36.058709 15438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:07:36 dist-test-slave-23bp krb5kdc[15210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561255, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.188@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
Sep 28 02:07:36 dist-test-slave-23bp krb5kdc[15210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.188@KRBTEST.COM
I20260928 02:07:36.080927 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:36.061937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54654 (local address 127.1.182.190:37209)
0928 02:07:36.062103 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:36.062107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:36.062909 (+ 802us) server_negotiation.cc:407] Connection header received
0928 02:07:36.065310 (+ 2401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:36.065315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:36.065385 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:36.065506 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:36.066743 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.067789 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.069792 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.070441 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.070597 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:36.074389 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:36.074397 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:36.074402 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:36.074448 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:36.077555 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.078435 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.078439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.078441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.078579 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.079130 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.079136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.079139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.079407 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:36.079526 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:36.080301 (+ 775us) server_negotiation.cc:299] Negotiation successful
0928 02:07:36.080488 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:07:36.082063 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:36.062010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:37209 (local address 127.0.0.1:54654)
0928 02:07:36.062749 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:36.062794 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:36.065012 (+ 2218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:36.065569 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:36.065575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:36.066243 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:36.066563 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:36.066581 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.067943 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.067949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:36.069636 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:36.069646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.070658 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.070663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:36.070913 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:36.071714 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:36.071743 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:36.074177 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:36.077677 (+ 3500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:36.077688 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:36.077701 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:36.078320 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:36.078698 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:36.078707 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:36.078714 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:36.078924 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:36.079539 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:36.079551 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:36.079959 (+ 408us) client_negotiation.cc:769] Sending connection context
0928 02:07:36.080201 (+ 242us) client_negotiation.cc:240] Negotiation successful
0928 02:07:36.080667 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":147,"spinlock_wait_cycles":17152,"thread_start_us":411,"threads_started":1}
I20260928 02:07:36.082552 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:36.059961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.1.182.188:46655)
0928 02:07:36.063587 (+ 3626us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:36.063599 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:36.063647 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:07:36.065012 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:36.065038 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:36.065339 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:36.065525 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:36.067230 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.069054 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.070776 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.071584 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.071715 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:36.076449 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:36.076461 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:36.076479 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:36.076525 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:36.079635 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.080258 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.080266 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.080268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.080423 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.080926 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.080931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.080933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.081442 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:36.081758 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:36.081917 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 02:07:36.082182 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":3378,"thread_start_us":3127,"threads_started":1}
I20260928 02:07:36.084070 15438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:07:36 dist-test-slave-23bp krb5kdc[15210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561255, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.188@KRBTEST.COM for kudu/127.1.182.189@KRBTEST.COM
Sep 28 02:07:36 dist-test-slave-23bp krb5kdc[15210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:07:36.098045 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:36.084719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38295 (local address 127.0.0.1:58764)
0928 02:07:36.084868 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:36.084888 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:36.084985 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:36.085349 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:36.085355 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:36.085781 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:36.085955 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:36.085967 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.087034 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.087041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:36.088174 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:36.088185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.088940 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.088946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:36.089178 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:36.089724 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:36.089748 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:36.091985 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:36.095647 (+ 3662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:36.095651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:36.095654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:36.096082 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:36.096577 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:36.096581 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:36.096583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:36.096721 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:36.097361 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:36.097365 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:36.097505 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 02:07:36.097620 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 02:07:36.097812 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 02:07:36.098363 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:36.084903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.1.182.189:38295)
0928 02:07:36.085081 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:36.085086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:36.085102 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:07:36.085150 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:36.085153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:36.085209 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:36.085290 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:36.086113 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.086888 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.088317 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.088801 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.088895 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:36.092188 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:36.092192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:36.092195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:36.092229 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:36.095519 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.096256 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.096262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.096266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.096431 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.096826 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.096833 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.096836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.097233 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:36.097385 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:36.097742 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 02:07:36.097945 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:07:36.099431 15438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:07:36.101591 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:36.085662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.1.182.190:37209)
0928 02:07:36.085812 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:36.085816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:36.085879 (+ 63us) server_negotiation.cc:407] Connection header received
0928 02:07:36.086061 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:36.086067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:36.086134 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:36.086238 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:36.087557 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.088333 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.090031 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.090425 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.090522 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:36.095346 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:36.095353 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:36.095357 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:36.095426 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:36.098709 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.099536 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.099540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.099544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.099695 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.100089 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.100094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.100098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.100451 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:36.100721 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:36.101016 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 02:07:36.101249 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 02:07:36 dist-test-slave-23bp krb5kdc[15210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561255, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.188@KRBTEST.COM for kudu/127.1.182.188@KRBTEST.COM
Sep 28 02:07:36 dist-test-slave-23bp krb5kdc[15210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.189@KRBTEST.COM
I20260928 02:07:36.113255 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:36.100159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:46655 (local address 127.0.0.1:39128)
0928 02:07:36.100340 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:36.100358 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:36.100460 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:36.100723 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:36.100727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:36.101106 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:36.101217 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:36.101224 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.102562 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.102572 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:36.103886 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:36.103897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.104565 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.104578 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:36.104756 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:36.105263 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:36.105302 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:36.107884 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:36.110844 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:36.110850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:36.110854 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:36.111264 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:36.111638 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:36.111642 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:36.111646 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:36.111777 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:36.112230 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:36.112234 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:36.112377 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 02:07:36.112500 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 02:07:36.112796 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 02:07:36.113534 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:36.100227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39128 (local address 127.1.182.188:46655)
0928 02:07:36.100463 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:36.100468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:36.100484 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:07:36.100556 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:36.100560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:36.100620 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:36.100709 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:36.101340 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.102360 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.104008 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.104406 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.104544 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:36.108044 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:36.108051 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:36.108055 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:36.108088 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:36.110703 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.111375 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.111383 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.111386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.111522 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.111879 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.111883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.111887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.112125 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:36.112268 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:36.112610 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 02:07:36.113189 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":357,"server-negotiator.queue_time_us":100}
I20260928 02:07:36.118024 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:36.103151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58770 (local address 127.1.182.189:38295)
0928 02:07:36.103385 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:36.103390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:36.103408 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:07:36.103482 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:36.103487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:36.103551 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:36.103634 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:36.104892 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.105969 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.107678 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.108074 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.108209 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:36.112314 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:36.112320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:36.112325 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:36.112362 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:36.115328 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.115955 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.115962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.115965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.116129 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.116602 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.116608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.116613 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.116948 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:36.117219 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:36.117478 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 02:07:36.117699 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260928 02:07:36.119330 15438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45: Bootstrap starting.
I20260928 02:07:36.123193 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:07:36.123222 15438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45: Neither blocks nor log segments found. Creating new log.
I20260928 02:07:36.124465 15438 log.cc:824] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45: Log is configured to *not* fsync() on all Append() calls
I20260928 02:07:36.127765 15438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45: No bootstrap required, opened a new log
I20260928 02:07:36.138836 15438 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } } peers { permanent_uuid: "69610fca01d64f868a63b7a228533e7f" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } } peers { permanent_uuid: "a3c5f5f3676a45268cdfba8b788fdc45" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } }
I20260928 02:07:36.139302 15438 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:07:36.139436 15438 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c5f5f3676a45268cdfba8b788fdc45, State: Initialized, Role: FOLLOWER
I20260928 02:07:36.140261 15438 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } } peers { permanent_uuid: "69610fca01d64f868a63b7a228533e7f" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } } peers { permanent_uuid: "a3c5f5f3676a45268cdfba8b788fdc45" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } }
I20260928 02:07:36.147326 15451 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } } peers { permanent_uuid: "69610fca01d64f868a63b7a228533e7f" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } } peers { permanent_uuid: "a3c5f5f3676a45268cdfba8b788fdc45" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } } }
I20260928 02:07:36.147768 15451 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:07:36.149513 15438 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:07:36.151598 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:36.128709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58784 (local address 127.1.182.189:38295)
0928 02:07:36.129028 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:36.129033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:36.130172 (+ 1139us) server_negotiation.cc:407] Connection header received
0928 02:07:36.130417 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:36.130422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:36.130500 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:36.130590 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:36.133457 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.134861 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.137931 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.138546 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.138668 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:36.143519 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:36.143531 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:36.143540 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:36.143607 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:36.148241 (+ 4634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.149098 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.149104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.149109 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.149266 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.149740 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.149745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.149747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.150109 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:36.150345 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:36.150727 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 02:07:36.151034 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260928 02:07:36.152083 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:36.128719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54674 (local address 127.1.182.190:37209)
0928 02:07:36.128891 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:36.128897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:36.129850 (+ 953us) server_negotiation.cc:407] Connection header received
0928 02:07:36.130131 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:36.130138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:36.130223 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:36.130291 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:36.132673 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.133778 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.135950 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.136581 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.136684 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:36.141332 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:36.141339 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:36.141343 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:36.141382 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:36.145753 (+ 4371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.146679 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.146685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.146690 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.146845 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.147352 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.147359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.147364 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.147878 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:36.148142 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:36.151428 (+ 3286us) server_negotiation.cc:299] Negotiation successful
0928 02:07:36.151679 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:07:36.154016 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:36.130673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39144 (local address 127.1.182.188:46655)
0928 02:07:36.130848 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:36.130855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:36.137721 (+ 6866us) server_negotiation.cc:407] Connection header received
0928 02:07:36.138035 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:36.138042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:36.138123 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:36.138204 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:36.140329 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.141405 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.143397 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.143820 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.143953 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:36.147031 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:36.147036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:36.147041 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:36.147109 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:36.150723 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.151699 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.151705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.151708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.151861 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.152305 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.152310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.152313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.152719 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:36.152958 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:36.153421 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 02:07:36.153659 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 02:07:36.166965 15462 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:07:36.167152 15462 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:07:36.162815 15371 master_runner.cc:186] Error getting master registration for 127.1.182.190:37209: OK, instance_id { permanent_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" instance_seqno: 1790561255032692 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:07:36.171175 15371 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:07:36.171255 15290 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } attempt: 4
Sep 28 02:07:36 dist-test-slave-23bp krb5kdc[15210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561254, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for kudu/127.1.182.188@KRBTEST.COM
I20260928 02:07:36.185459 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:36.171853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:46655 (local address 127.0.0.1:39150)
0928 02:07:36.172123 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:36.172142 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:36.172247 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:36.172766 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:36.172772 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:36.173273 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:36.173495 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:36.173511 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.174696 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.174703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:36.175693 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:36.175704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.176377 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.176383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:36.176565 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:36.177157 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:36.177174 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:36.179413 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:36.182965 (+ 3552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:36.182971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:36.182975 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:36.183420 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:36.183797 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:36.183802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:36.183805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:36.183942 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:36.184476 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:36.184481 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:36.184641 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 02:07:36.184771 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 02:07:36.185054 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260928 02:07:36.185452 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:36.172088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39150 (local address 127.1.182.188:46655)
0928 02:07:36.172243 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:36.172247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:36.172264 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:07:36.172380 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:36.172386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:36.172462 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:36.172563 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:36.173682 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.174559 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.175825 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.176241 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.176368 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:36.179610 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:36.179615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:36.179618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:36.179647 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:36.182833 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.183547 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.183552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.183554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.183681 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.184051 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.184056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.184067 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.184357 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:36.184510 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:36.184876 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 02:07:36.185081 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:07:36.190506 15290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c: Bootstrap starting.
I20260928 02:07:36.193758 15290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c: Neither blocks nor log segments found. Creating new log.
I20260928 02:07:36.194861 15290 log.cc:824] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:07:36.198171 15290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c: No bootstrap required, opened a new log
I20260928 02:07:36.205821 15290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } } peers { permanent_uuid: "69610fca01d64f868a63b7a228533e7f" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } } peers { permanent_uuid: "a3c5f5f3676a45268cdfba8b788fdc45" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } }
I20260928 02:07:36.206130 15290 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:07:36.206286 15290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b6ec1fd6f44b0d839c38f840ff0d7c, State: Initialized, Role: FOLLOWER
I20260928 02:07:36.206876 15290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } } peers { permanent_uuid: "69610fca01d64f868a63b7a228533e7f" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } } peers { permanent_uuid: "a3c5f5f3676a45268cdfba8b788fdc45" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } }
I20260928 02:07:36.207969 15465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } } peers { permanent_uuid: "69610fca01d64f868a63b7a228533e7f" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } } peers { permanent_uuid: "a3c5f5f3676a45268cdfba8b788fdc45" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } } }
I20260928 02:07:36.208362 15465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:07:36.209177 15290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:07:36.221494 15476 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:07:36.221634 15476 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:07:36.220610 15223 master_runner.cc:186] Error getting master registration for 127.1.182.189:38295: OK, instance_id { permanent_uuid: "69610fca01d64f868a63b7a228533e7f" instance_seqno: 1790561255538416 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:07:36.232679 15223 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:07:36.266350 15364 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } attempt: 5
W20260928 02:07:36.272580 15244 tablet.cc:2369] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:07:36 dist-test-slave-23bp krb5kdc[15210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561255, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.189@KRBTEST.COM for kudu/127.1.182.188@KRBTEST.COM
W20260928 02:07:36.278838 15392 tablet.cc:2369] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:07:36.283190 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:36.266982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:46655 (local address 127.0.0.1:39160)
0928 02:07:36.267219 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:36.267239 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:36.267359 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:36.267792 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:36.267798 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:36.268354 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:36.268569 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:36.268581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.270084 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.270090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:36.271614 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:36.271630 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.274022 (+ 2392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.274028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:36.274278 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:36.274935 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:36.274955 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:36.277051 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:36.280768 (+ 3717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:36.280773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:36.280777 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:36.281161 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:36.281532 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:36.281538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:36.281543 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:36.281673 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:36.282181 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:36.282186 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:36.282334 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 02:07:36.282478 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 02:07:36.282774 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260928 02:07:36.283201 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:36.267123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39160 (local address 127.1.182.188:46655)
0928 02:07:36.267319 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:36.267324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:36.267346 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:07:36.267505 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:36.267511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:36.267596 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:36.267712 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:36.268730 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.269932 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.271800 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:36.272515 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:36.272668 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:36.277256 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:36.277261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:36.277265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:36.277322 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:36.280632 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.281292 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.281296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.281299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.281423 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:36.281788 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:36.281792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:36.281794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:36.282065 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:36.282219 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:36.282594 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 02:07:36.282816 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:07:36.287846 15364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f: Bootstrap starting.
I20260928 02:07:36.291811 15364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f: Neither blocks nor log segments found. Creating new log.
I20260928 02:07:36.292984 15364 log.cc:824] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:07:36.295708 15364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f: No bootstrap required, opened a new log
I20260928 02:07:36.303529 15364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } } peers { permanent_uuid: "69610fca01d64f868a63b7a228533e7f" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } } peers { permanent_uuid: "a3c5f5f3676a45268cdfba8b788fdc45" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } }
I20260928 02:07:36.303867 15364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:07:36.303979 15364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69610fca01d64f868a63b7a228533e7f, State: Initialized, Role: FOLLOWER
I20260928 02:07:36.304831 15364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } } peers { permanent_uuid: "69610fca01d64f868a63b7a228533e7f" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } } peers { permanent_uuid: "a3c5f5f3676a45268cdfba8b788fdc45" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } }
I20260928 02:07:36.305882 15480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } } peers { permanent_uuid: "69610fca01d64f868a63b7a228533e7f" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } } peers { permanent_uuid: "a3c5f5f3676a45268cdfba8b788fdc45" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } } }
I20260928 02:07:36.306711 15465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:36.307022 15465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } } peers { permanent_uuid: "69610fca01d64f868a63b7a228533e7f" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } } peers { permanent_uuid: "a3c5f5f3676a45268cdfba8b788fdc45" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } }
I20260928 02:07:36.307328 15364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:07:36.308947 15465 leader_election.cc:288] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69610fca01d64f868a63b7a228533e7f (127.1.182.189:38295), a3c5f5f3676a45268cdfba8b788fdc45 (127.1.182.188:46655)
I20260928 02:07:36.310626 15339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69610fca01d64f868a63b7a228533e7f" is_pre_election: true
I20260928 02:07:36.311264 15339 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72b6ec1fd6f44b0d839c38f840ff0d7c in term 0.
I20260928 02:07:36.316154 15480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:07:36.316592 15242 leader_election.cc:302] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69610fca01d64f868a63b7a228533e7f, 72b6ec1fd6f44b0d839c38f840ff0d7c; no voters:
I20260928 02:07:36.317039 15465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:07:36.317236 15465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:07:36.317355 15465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:07:36.314750 15413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c5f5f3676a45268cdfba8b788fdc45" is_pre_election: true
I20260928 02:07:36.317768 15413 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72b6ec1fd6f44b0d839c38f840ff0d7c in term 0.
I20260928 02:07:36.322621 15465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } } peers { permanent_uuid: "69610fca01d64f868a63b7a228533e7f" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } } peers { permanent_uuid: "a3c5f5f3676a45268cdfba8b788fdc45" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } }
I20260928 02:07:36.324126 15339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69610fca01d64f868a63b7a228533e7f"
I20260928 02:07:36.324390 15339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:07:36.328780 15339 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72b6ec1fd6f44b0d839c38f840ff0d7c in term 1.
I20260928 02:07:36.329424 15242 leader_election.cc:302] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69610fca01d64f868a63b7a228533e7f, 72b6ec1fd6f44b0d839c38f840ff0d7c; no voters:
I20260928 02:07:36.334328 15413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c5f5f3676a45268cdfba8b788fdc45"
I20260928 02:07:36.334642 15413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:07:36.342180 15413 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72b6ec1fd6f44b0d839c38f840ff0d7c in term 1.
I20260928 02:07:36.344149 15465 leader_election.cc:288] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [CANDIDATE]: Term 1 election: Requested vote from peers 69610fca01d64f868a63b7a228533e7f (127.1.182.189:38295), a3c5f5f3676a45268cdfba8b788fdc45 (127.1.182.188:46655)
I20260928 02:07:36.344666 15465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:07:36.345181 15465 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [term 1 LEADER]: Becoming Leader. State: Replica: 72b6ec1fd6f44b0d839c38f840ff0d7c, State: Running, Role: LEADER
I20260928 02:07:36.346163 15465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } } peers { permanent_uuid: "69610fca01d64f868a63b7a228533e7f" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } } peers { permanent_uuid: "a3c5f5f3676a45268cdfba8b788fdc45" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } }
I20260928 02:07:36.358022 15485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72b6ec1fd6f44b0d839c38f840ff0d7c. Latest consensus state: current_term: 1 leader_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } } peers { permanent_uuid: "69610fca01d64f868a63b7a228533e7f" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } } peers { permanent_uuid: "a3c5f5f3676a45268cdfba8b788fdc45" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } } }
I20260928 02:07:36.358749 15485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [sys.catalog]: This master's current role is: LEADER
I20260928 02:07:36.370678 15489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:07:36.373956 15489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:07:36.393508 15495 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:07:36.396453 15413 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45 [term 1 FOLLOWER]: Refusing update from remote peer 72b6ec1fd6f44b0d839c38f840ff0d7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:07:36.397622 15485 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3c5f5f3676a45268cdfba8b788fdc45" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:07:36.413450 15339 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f [term 1 FOLLOWER]: Refusing update from remote peer 72b6ec1fd6f44b0d839c38f840ff0d7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:07:36.414546 15485 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "69610fca01d64f868a63b7a228533e7f" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:07:36.422695 15451 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72b6ec1fd6f44b0d839c38f840ff0d7c. Latest consensus state: current_term: 1 leader_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } } peers { permanent_uuid: "69610fca01d64f868a63b7a228533e7f" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } } peers { permanent_uuid: "a3c5f5f3676a45268cdfba8b788fdc45" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } } }
I20260928 02:07:36.423331 15451 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:07:36.442198 15493 mvcc.cc:205] Tried to move back new op lower bound from 7334138906185973760 to 7334138906009878528. Current Snapshot: MvccSnapshot[applied={T|T < 7334138906185973760}]
I20260928 02:07:36.458712 15465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } } peers { permanent_uuid: "69610fca01d64f868a63b7a228533e7f" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } } peers { permanent_uuid: "a3c5f5f3676a45268cdfba8b788fdc45" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } } }
I20260928 02:07:36.459481 15465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [sys.catalog]: This master's current role is: LEADER
I20260928 02:07:36.461931 15489 catalog_manager.cc:1347] Generated new cluster ID: 11071deec7614ea2bc2b5c50ab29b2f3
I20260928 02:07:36.462073 15489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:07:36.463577 15451 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } } peers { permanent_uuid: "69610fca01d64f868a63b7a228533e7f" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } } peers { permanent_uuid: "a3c5f5f3676a45268cdfba8b788fdc45" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } } }
I20260928 02:07:36.463809 15451 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:07:36.476223 15480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72b6ec1fd6f44b0d839c38f840ff0d7c. Latest consensus state: current_term: 1 leader_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } } peers { permanent_uuid: "69610fca01d64f868a63b7a228533e7f" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } } peers { permanent_uuid: "a3c5f5f3676a45268cdfba8b788fdc45" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } } }
I20260928 02:07:36.476521 15480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:07:36.478317 15465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } } peers { permanent_uuid: "69610fca01d64f868a63b7a228533e7f" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } } peers { permanent_uuid: "a3c5f5f3676a45268cdfba8b788fdc45" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } } }
I20260928 02:07:36.478700 15465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c [sys.catalog]: This master's current role is: LEADER
I20260928 02:07:36.482760 15480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72b6ec1fd6f44b0d839c38f840ff0d7c" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37209 } } peers { permanent_uuid: "69610fca01d64f868a63b7a228533e7f" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38295 } } peers { permanent_uuid: "a3c5f5f3676a45268cdfba8b788fdc45" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 46655 } } }
I20260928 02:07:36.483016 15480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:07:36.488209 15489 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:07:36.490433 15489 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:07:36.504999 15489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72b6ec1fd6f44b0d839c38f840ff0d7c: Generated new TSK 0
I20260928 02:07:36.506664 15489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:07:37.169200 15462 catalog_manager.cc:1259] Loaded cluster ID: 11071deec7614ea2bc2b5c50ab29b2f3
I20260928 02:07:37.169365 15462 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45: loading cluster ID for follower catalog manager: success
I20260928 02:07:37.172076 15462 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45: acquiring CA information for follower catalog manager: success
I20260928 02:07:37.174245 15462 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a3c5f5f3676a45268cdfba8b788fdc45: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:07:37.397974 15495 catalog_manager.cc:1259] Loaded cluster ID: 11071deec7614ea2bc2b5c50ab29b2f3
I20260928 02:07:37.398078 15495 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f: loading cluster ID for follower catalog manager: success
I20260928 02:07:37.400602 15495 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f: acquiring CA information for follower catalog manager: success
I20260928 02:07:37.402494 15495 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 69610fca01d64f868a63b7a228533e7f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:07:37.604096 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 15223
I20260928 02:07:37.605217 15285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:07:37.716161 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 15223
I20260928 02:07:37.747196 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 15297
I20260928 02:07:37.748426 15359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:07:37.853890 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 15297
I20260928 02:07:37.876047 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 15371
I20260928 02:07:37.877429 15433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:07:37.985170 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 15371
2026-09-28T02:07:37Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4763 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:07:38 dist-test-slave-23bp krb5kdc[15515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:07:38 dist-test-slave-23bp krb5kdc[15515](info): set up 2 sockets
Sep 28 02:07:38 dist-test-slave-23bp krb5kdc[15515](info): commencing operation
krb5kdc: starting...
W20260928 02:07:39.185990 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.008s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:07:39 dist-test-slave-23bp krb5kdc[15515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:07:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:07:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:07:39.424185 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:43899
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:33081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:43899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:07:39.759918 15528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:39.760221 15528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:39.760308 15528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:39.767536 15528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:07:39.767642 15528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:39.767688 15528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:39.767772 15528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:07:39.767822 15528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:07:39.777968 15528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:33081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:43899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:43899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:39.779306 15528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:39.780582 15528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:39.792157 15536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:39.792151 15534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:39.792163 15533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:39.793408 15528 server_base.cc:1034] running on GCE node
I20260928 02:07:39.794196 15528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:39.796029 15528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:39.797272 15528 hybrid_clock.cc:648] HybridClock initialized: now 1790561259797204 us; error 73 us; skew 500 ppm
I20260928 02:07:39.797636 15528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:39 dist-test-slave-23bp krb5kdc[15515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561259, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:39.800822 15528 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:07:39.802349 15528 webserver.cc:460] Webserver started at http://127.1.182.190:44441/ using document root <none> and password file <none>
I20260928 02:07:39.802986 15528 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:39.803155 15528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:07:39.803435 15528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:07:39.806334 15528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "5dbe674a0fae4fc29f031c4d3663ff0e"
format_stamp: "Formatted at 2026-09-28 02:07:39 on dist-test-slave-23bp"
server_key: "607ffe08fd4b267044e10382d44a0f83"
server_key_iv: "b6324f5fcc98390a29d2d2dafda702d8"
server_key_version: "encryptionkey@0"
I20260928 02:07:39.807022 15528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "5dbe674a0fae4fc29f031c4d3663ff0e"
format_stamp: "Formatted at 2026-09-28 02:07:39 on dist-test-slave-23bp"
server_key: "607ffe08fd4b267044e10382d44a0f83"
server_key_iv: "b6324f5fcc98390a29d2d2dafda702d8"
server_key_version: "encryptionkey@0"
I20260928 02:07:39.812184 15528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:07:39.815748 15543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:39.817071 15528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:07:39.817229 15528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "5dbe674a0fae4fc29f031c4d3663ff0e"
format_stamp: "Formatted at 2026-09-28 02:07:39 on dist-test-slave-23bp"
server_key: "607ffe08fd4b267044e10382d44a0f83"
server_key_iv: "b6324f5fcc98390a29d2d2dafda702d8"
server_key_version: "encryptionkey@0"
I20260928 02:07:39.817389 15528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:39.832701 15528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:39.837126 15528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:39.837353 15528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:39.856135 15528 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:43899
I20260928 02:07:39.856142 15595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:43899 every 8 connection(s)
I20260928 02:07:39.857540 15528 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:07:39.861264 15596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:07:39.867869 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 15528
I20260928 02:07:39.868078 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:07:39.868714 1754 external_mini_cluster.cc:1428] Setting key 4a55d422d7610c5a6ecb29a8fe6025a9
I20260928 02:07:39.871824 15596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e: Bootstrap starting.
I20260928 02:07:39.875379 15596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e: Neither blocks nor log segments found. Creating new log.
I20260928 02:07:39.876533 15596 log.cc:824] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:07:39.879870 15596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e: No bootstrap required, opened a new log
Sep 28 02:07:39 dist-test-slave-23bp krb5kdc[15515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:07:39.889178 15596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbe674a0fae4fc29f031c4d3663ff0e" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 43899 } }
I20260928 02:07:39.889575 15596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:07:39.889698 15596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dbe674a0fae4fc29f031c4d3663ff0e, State: Initialized, Role: FOLLOWER
I20260928 02:07:39.890393 15596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbe674a0fae4fc29f031c4d3663ff0e" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 43899 } }
I20260928 02:07:39.890692 15596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:07:39.890843 15596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:07:39.891048 15596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:07:39.893380 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:39.870542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44970 (local address 127.1.182.190:43899)
0928 02:07:39.871387 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:39.871400 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:39.871443 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:07:39.873309 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:39.873334 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:39.873715 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:39.874030 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:39.875951 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:39.877613 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:39.879596 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:39.880420 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:39.880570 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:39.885536 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:39.885544 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:39.885559 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:39.885600 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:39.888944 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:39.889723 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:39.889730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:39.889736 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:39.889962 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:39.890433 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:39.890436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:39.890438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:39.891202 (+ 764us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:39.891446 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:39.891913 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 02:07:39.892352 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":397,"threads_started":1}
I20260928 02:07:39.893836 15596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbe674a0fae4fc29f031c4d3663ff0e" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 43899 } }
I20260928 02:07:39.894321 15596 leader_election.cc:302] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dbe674a0fae4fc29f031c4d3663ff0e; no voters:
I20260928 02:07:39.895036 15596 leader_election.cc:288] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:07:39.895201 15601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:07:39.897475 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:07:39.897621 15596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:07:39.901921 15601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e [term 1 LEADER]: Becoming Leader. State: Replica: 5dbe674a0fae4fc29f031c4d3663ff0e, State: Running, Role: LEADER
I20260928 02:07:39.902590 15601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbe674a0fae4fc29f031c4d3663ff0e" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 43899 } }
I20260928 02:07:39.905699 15602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5dbe674a0fae4fc29f031c4d3663ff0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbe674a0fae4fc29f031c4d3663ff0e" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 43899 } } }
I20260928 02:07:39.906083 15602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e [sys.catalog]: This master's current role is: LEADER
I20260928 02:07:39.906812 15603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dbe674a0fae4fc29f031c4d3663ff0e. Latest consensus state: current_term: 1 leader_uuid: "5dbe674a0fae4fc29f031c4d3663ff0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbe674a0fae4fc29f031c4d3663ff0e" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 43899 } } }
I20260928 02:07:39.907199 15603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e [sys.catalog]: This master's current role is: LEADER
I20260928 02:07:39.907675 15612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:07:39.912341 15612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:07:39.920569 15612 catalog_manager.cc:1347] Generated new cluster ID: 8e6fc5242cae4472952cd73595eeae05
I20260928 02:07:39.920706 15612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:07:39.930220 15612 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:07:39.932176 15612 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:07:39.948752 15612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dbe674a0fae4fc29f031c4d3663ff0e: Generated new TSK 0
I20260928 02:07:39.949922 15612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:07:39.958276 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 15528
I20260928 02:07:39.959496 15591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:07:40.049674 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 15528
2026-09-28T02:07:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2059 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:07:40 dist-test-slave-23bp krb5kdc[15624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:07:40 dist-test-slave-23bp krb5kdc[15624](info): set up 2 sockets
Sep 28 02:07:40 dist-test-slave-23bp krb5kdc[15624](info): commencing operation
krb5kdc: starting...
W20260928 02:07:41.141849 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:07:41 dist-test-slave-23bp krb5kdc[15624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:07:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:07:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:07:41.374841 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:34479
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:34479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:07:41.698734 15638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:41.699014 15638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:41.699187 15638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:41.706858 15638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:07:41.706955 15638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:41.707042 15638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:41.707190 15638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:07:41.707268 15638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:07:41.716924 15638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:34479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:34479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:41.718197 15638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:41.719683 15638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:41.730916 15643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:41.731003 15644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:41.731307 15646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:41.732429 15638 server_base.cc:1034] running on GCE node
I20260928 02:07:41.733155 15638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:41.734491 15638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:41.735684 15638 hybrid_clock.cc:648] HybridClock initialized: now 1790561261735644 us; error 36 us; skew 500 ppm
I20260928 02:07:41.736011 15638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:41 dist-test-slave-23bp krb5kdc[15624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561261, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:41.738966 15638 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:07:41.740372 15638 webserver.cc:460] Webserver started at http://127.1.182.190:45133/ using document root <none> and password file <none>
I20260928 02:07:41.741022 15638 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:41.741166 15638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:07:41.741449 15638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:07:41.744367 15638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "317c6ee7d88947eb8b8ae4731321b223"
format_stamp: "Formatted at 2026-09-28 02:07:41 on dist-test-slave-23bp"
server_key: "b63465a1e8561cce45bee503641a6be1"
server_key_iv: "fffdc50cad54da34f43e612c993ab124"
server_key_version: "encryptionkey@0"
I20260928 02:07:41.745126 15638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "317c6ee7d88947eb8b8ae4731321b223"
format_stamp: "Formatted at 2026-09-28 02:07:41 on dist-test-slave-23bp"
server_key: "b63465a1e8561cce45bee503641a6be1"
server_key_iv: "fffdc50cad54da34f43e612c993ab124"
server_key_version: "encryptionkey@0"
I20260928 02:07:41.749974 15638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:07:41.753247 15653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:41.754493 15638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:07:41.754673 15638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "317c6ee7d88947eb8b8ae4731321b223"
format_stamp: "Formatted at 2026-09-28 02:07:41 on dist-test-slave-23bp"
server_key: "b63465a1e8561cce45bee503641a6be1"
server_key_iv: "fffdc50cad54da34f43e612c993ab124"
server_key_version: "encryptionkey@0"
I20260928 02:07:41.754856 15638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:41.782944 15638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:41.787305 15638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:41.787647 15638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:41.805732 15638 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:34479
I20260928 02:07:41.805764 15704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:34479 every 8 connection(s)
I20260928 02:07:41.807056 15638 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:07:41.810734 15705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:07:41.816444 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 15638
I20260928 02:07:41.816660 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:07:41.817312 1754 external_mini_cluster.cc:1428] Setting key 9c1e4f8bc27c36e46f94cf294e3041cb
I20260928 02:07:41.822150 15705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223: Bootstrap starting.
I20260928 02:07:41.826004 15705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223: Neither blocks nor log segments found. Creating new log.
I20260928 02:07:41.827297 15705 log.cc:824] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223: Log is configured to *not* fsync() on all Append() calls
I20260928 02:07:41.830523 15705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223: No bootstrap required, opened a new log
Sep 28 02:07:41 dist-test-slave-23bp krb5kdc[15624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:07:41.839301 15705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317c6ee7d88947eb8b8ae4731321b223" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 34479 } }
I20260928 02:07:41.839713 15705 raft_consensus.cc:374] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:07:41.839814 15705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 317c6ee7d88947eb8b8ae4731321b223, State: Initialized, Role: FOLLOWER
I20260928 02:07:41.840564 15705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317c6ee7d88947eb8b8ae4731321b223" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 34479 } }
I20260928 02:07:41.840863 15705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:07:41.841009 15705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:07:41.841221 15705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:07:41.842203 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:41.819449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49482 (local address 127.1.182.190:34479)
0928 02:07:41.820230 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:41.820247 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:41.820295 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:07:41.822621 (+ 2326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:41.822650 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:41.823165 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:41.823585 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:41.824890 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:41.826883 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:41.828700 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:41.829704 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:41.829886 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:41.834391 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:41.834399 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:41.834417 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:41.834453 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:41.837754 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:41.838656 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:41.838663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:41.838669 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:41.838829 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:41.839263 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:41.839266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:41.839268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:41.839968 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:41.840259 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:41.840640 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 02:07:41.841060 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":337,"threads_started":1}
I20260928 02:07:41.843770 15705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317c6ee7d88947eb8b8ae4731321b223" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 34479 } }
I20260928 02:07:41.844278 15705 leader_election.cc:302] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 317c6ee7d88947eb8b8ae4731321b223; no voters:
I20260928 02:07:41.844969 15705 leader_election.cc:288] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:07:41.845088 15710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:07:41.845855 15710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 1 LEADER]: Becoming Leader. State: Replica: 317c6ee7d88947eb8b8ae4731321b223, State: Running, Role: LEADER
I20260928 02:07:41.846393 15710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317c6ee7d88947eb8b8ae4731321b223" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 34479 } }
I20260928 02:07:41.847054 15705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:07:41.849294 15712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 317c6ee7d88947eb8b8ae4731321b223. Latest consensus state: current_term: 1 leader_uuid: "317c6ee7d88947eb8b8ae4731321b223" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317c6ee7d88947eb8b8ae4731321b223" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 34479 } } }
I20260928 02:07:41.849771 15712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [sys.catalog]: This master's current role is: LEADER
I20260928 02:07:41.851280 15719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:07:41.851850 15711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "317c6ee7d88947eb8b8ae4731321b223" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317c6ee7d88947eb8b8ae4731321b223" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 34479 } } }
I20260928 02:07:41.852165 15711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [sys.catalog]: This master's current role is: LEADER
I20260928 02:07:41.856427 15719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:07:41.865944 15719 catalog_manager.cc:1347] Generated new cluster ID: fe7d6cb0497a4cd7aeee849a06f00b76
I20260928 02:07:41.866083 15719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:07:41.887936 15719 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:07:41.889326 15719 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:07:41.906550 15719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223: Generated new TSK 0
I20260928 02:07:41.907503 15719 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.182.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
I20260928 02:07:41.980135 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:34479
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:07:42.337206 15731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:42.337466 15731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:42.337559 15731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:42.344846 15731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:42.344949 15731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:42.345114 15731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:07:42.354890 15731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:34479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:42.356169 15731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:42.357491 15731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:42.370288 15736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:42.370324 15739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:42.370378 15737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:42.371491 15731 server_base.cc:1034] running on GCE node
I20260928 02:07:42.372115 15731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:42.373032 15731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:42.374269 15731 hybrid_clock.cc:648] HybridClock initialized: now 1790561262374229 us; error 54 us; skew 500 ppm
I20260928 02:07:42.374590 15731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:42 dist-test-slave-23bp krb5kdc[15624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561262, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:42.377800 15731 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:07:42.379396 15731 webserver.cc:460] Webserver started at http://127.1.182.129:32799/ using document root <none> and password file <none>
I20260928 02:07:42.380168 15731 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:42.380328 15731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:07:42.380638 15731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:07:42.383662 15731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "814c38565bcd48eeb9de69fe0b86f6ea"
format_stamp: "Formatted at 2026-09-28 02:07:42 on dist-test-slave-23bp"
server_key: "c259bc7e052e007f91ada4900e261488"
server_key_iv: "44c684cfbf17e6d7767c0646e6210fb3"
server_key_version: "encryptionkey@0"
I20260928 02:07:42.384418 15731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "814c38565bcd48eeb9de69fe0b86f6ea"
format_stamp: "Formatted at 2026-09-28 02:07:42 on dist-test-slave-23bp"
server_key: "c259bc7e052e007f91ada4900e261488"
server_key_iv: "44c684cfbf17e6d7767c0646e6210fb3"
server_key_version: "encryptionkey@0"
I20260928 02:07:42.389447 15731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 02:07:42.392812 15746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:42.394043 15731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:07:42.394229 15731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "814c38565bcd48eeb9de69fe0b86f6ea"
format_stamp: "Formatted at 2026-09-28 02:07:42 on dist-test-slave-23bp"
server_key: "c259bc7e052e007f91ada4900e261488"
server_key_iv: "44c684cfbf17e6d7767c0646e6210fb3"
server_key_version: "encryptionkey@0"
I20260928 02:07:42.394426 15731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:42.409571 15731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:42.413498 15731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:42.413779 15731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:42.414685 15731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:07:42.416031 15731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:07:42.416110 15731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:42.416268 15731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:07:42.416343 15731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:42.447876 15731 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:34723
I20260928 02:07:42.447942 15859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:34723 every 8 connection(s)
I20260928 02:07:42.449682 15731 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:07:42.459537 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 15731
I20260928 02:07:42.459769 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:07:42.460438 1754 external_mini_cluster.cc:1428] Setting key e87396542f042a55bb878eba240c3ea2
Sep 28 02:07:42 dist-test-slave-23bp krb5kdc[15624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561262, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:07:42.469774 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:42.452418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:51851 (local address 127.1.182.190:34479)
0928 02:07:42.452897 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:42.452901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:42.453836 (+ 935us) server_negotiation.cc:407] Connection header received
0928 02:07:42.455751 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:42.455757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:42.455847 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:42.455965 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:42.457460 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:42.458214 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:42.459925 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:42.460316 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:42.460459 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:42.464132 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:42.464137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:42.464140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:42.464171 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:42.466712 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:42.467492 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:42.467501 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:42.467504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:42.467635 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:42.468059 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:42.468068 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:42.468070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:42.468324 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:42.468477 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:42.469173 (+ 696us) server_negotiation.cc:299] Negotiation successful
0928 02:07:42.469400 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":271,"threads_started":1}
I20260928 02:07:42.470717 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:42.452944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:51851)
0928 02:07:42.453675 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:42.453719 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:42.455471 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:42.456206 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:42.456214 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:42.456713 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:42.457295 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:42.457312 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:42.458380 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:42.458385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:42.459746 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:42.459758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:42.460551 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:42.460559 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:42.460806 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:42.461556 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:42.461578 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:42.463905 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:42.466854 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:42.466864 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:42.466881 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:42.467370 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:42.467745 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:42.467751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:42.467755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:42.467942 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:42.468462 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:42.468471 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:42.468808 (+ 337us) client_negotiation.cc:769] Sending connection context
0928 02:07:42.469072 (+ 264us) client_negotiation.cc:240] Negotiation successful
0928 02:07:42.469623 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":301,"threads_started":1}
I20260928 02:07:42.472280 15860 heartbeater.cc:347] Connected to a master server at 127.1.182.190:34479
I20260928 02:07:42.472574 15860 heartbeater.cc:464] Registering TS with master...
I20260928 02:07:42.473284 15860 heartbeater.cc:511] Master 127.1.182.190:34479 requested a full tablet report, sending...
I20260928 02:07:42.475411 15670 ts_manager.cc:194] Registered new tserver with Master: 814c38565bcd48eeb9de69fe0b86f6ea (127.1.182.129:34723)
I20260928 02:07:42.476800 15670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:51851
WARNING: no policy specified for kudu/127.1.182.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
I20260928 02:07:42.511330 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:34479
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:07:42.834864 15866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:42.835251 15866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:42.835446 15866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:42.843374 15866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:42.843513 15866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:42.843706 15866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:07:42.853989 15866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:34479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:42.855432 15866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:42.856789 15866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:42.870606 15872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:42.870657 15874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:42.870695 15871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:42.871516 15866 server_base.cc:1034] running on GCE node
I20260928 02:07:42.872181 15866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:42.873088 15866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:42.874295 15866 hybrid_clock.cc:648] HybridClock initialized: now 1790561262874256 us; error 26 us; skew 500 ppm
I20260928 02:07:42.874593 15866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:42 dist-test-slave-23bp krb5kdc[15624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561262, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:42.877854 15866 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:07:42.879412 15866 webserver.cc:460] Webserver started at http://127.1.182.130:46751/ using document root <none> and password file <none>
I20260928 02:07:42.880137 15866 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:42.880327 15866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:07:42.880666 15866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:07:42.883759 15866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "305b82159a2e4591b5d7597a4620846c"
format_stamp: "Formatted at 2026-09-28 02:07:42 on dist-test-slave-23bp"
server_key: "8ecac998a08a48ea267e1260040402e1"
server_key_iv: "149eee9f057f5584d28d400f2c307e3d"
server_key_version: "encryptionkey@0"
I20260928 02:07:42.884542 15866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "305b82159a2e4591b5d7597a4620846c"
format_stamp: "Formatted at 2026-09-28 02:07:42 on dist-test-slave-23bp"
server_key: "8ecac998a08a48ea267e1260040402e1"
server_key_iv: "149eee9f057f5584d28d400f2c307e3d"
server_key_version: "encryptionkey@0"
I20260928 02:07:42.890348 15866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:07:42.893882 15881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:42.895378 15866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:07:42.895583 15866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "305b82159a2e4591b5d7597a4620846c"
format_stamp: "Formatted at 2026-09-28 02:07:42 on dist-test-slave-23bp"
server_key: "8ecac998a08a48ea267e1260040402e1"
server_key_iv: "149eee9f057f5584d28d400f2c307e3d"
server_key_version: "encryptionkey@0"
I20260928 02:07:42.895799 15866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:42.907883 15866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:42.912634 15866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:42.912952 15866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:42.913945 15866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:07:42.915458 15866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:07:42.915580 15866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:42.915748 15866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:07:42.915829 15866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:42.951514 15866 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:45131
I20260928 02:07:42.951632 15994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:45131 every 8 connection(s)
I20260928 02:07:42.953101 15866 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:07:42.953409 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 15866
I20260928 02:07:42.953624 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:07:42.954281 1754 external_mini_cluster.cc:1428] Setting key a4e0e3b28aa062c00c54384a2e2e28cb
Sep 28 02:07:42 dist-test-slave-23bp krb5kdc[15624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561262, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:07:42.976698 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:42.956362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:43091 (local address 127.1.182.190:34479)
0928 02:07:42.956542 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:42.956546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:42.958093 (+ 1547us) server_negotiation.cc:407] Connection header received
0928 02:07:42.960267 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:42.960276 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:42.960364 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:42.960487 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:42.962033 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:42.963213 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:42.965417 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:42.966144 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:42.966297 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:42.970741 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:42.970745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:42.970749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:42.970782 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:42.973450 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:42.974211 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:42.974217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:42.974221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:42.974370 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:42.974815 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:42.974822 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:42.974826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:42.975215 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:42.975394 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:42.976100 (+ 706us) server_negotiation.cc:299] Negotiation successful
0928 02:07:42.976336 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:07:42.977722 15997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:42.956993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.130:43091)
0928 02:07:42.957900 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:42.957959 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:42.959970 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:42.960752 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:42.960761 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:42.961277 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:42.961845 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:42.961863 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:42.963360 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:42.963363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:42.965222 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:42.965240 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:42.966407 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:42.966417 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:42.966778 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:42.967562 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:42.967596 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:42.970468 (+ 2872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:42.973606 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:42.973616 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:42.973635 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:42.974076 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:42.974486 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:42.974492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:42.974495 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:42.974696 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:42.975355 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:42.975363 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:42.975730 (+ 367us) client_negotiation.cc:769] Sending connection context
0928 02:07:42.976014 (+ 284us) client_negotiation.cc:240] Negotiation successful
0928 02:07:42.976572 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":384,"threads_started":1}
WARNING: no policy specified for kudu/127.1.182.131@KRBTEST.COM; defaulting to no policy
I20260928 02:07:42.979259 15995 heartbeater.cc:347] Connected to a master server at 127.1.182.190:34479
I20260928 02:07:42.979645 15995 heartbeater.cc:464] Registering TS with master...
I20260928 02:07:42.980381 15995 heartbeater.cc:511] Master 127.1.182.190:34479 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.131@KRBTEST.COM" created.
I20260928 02:07:42.982056 15670 ts_manager.cc:194] Registered new tserver with Master: 305b82159a2e4591b5d7597a4620846c (127.1.182.130:45131)
I20260928 02:07:42.982897 15670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:43091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
I20260928 02:07:43.011435 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:34479
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:07:43.363471 16000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:43.363736 16000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:43.363905 16000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:43.371060 16000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:43.371227 16000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:43.371423 16000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:07:43.381323 16000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:34479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:43.382753 16000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:43.384146 16000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:43.397045 16006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:43.397101 16008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:43.397125 16005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:43.397974 16000 server_base.cc:1034] running on GCE node
I20260928 02:07:43.398600 16000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:43.399629 16000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:43.400852 16000 hybrid_clock.cc:648] HybridClock initialized: now 1790561263400785 us; error 66 us; skew 500 ppm
I20260928 02:07:43.401265 16000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:43 dist-test-slave-23bp krb5kdc[15624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561263, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:43.404439 16000 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:07:43.405808 16000 webserver.cc:460] Webserver started at http://127.1.182.131:37413/ using document root <none> and password file <none>
I20260928 02:07:43.406813 16000 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:43.406973 16000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:07:43.407289 16000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:07:43.410331 16000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c"
format_stamp: "Formatted at 2026-09-28 02:07:43 on dist-test-slave-23bp"
server_key: "1cde41b6dbe852885c406b9a222d602c"
server_key_iv: "f15474c4753775cd2242df0521109fe1"
server_key_version: "encryptionkey@0"
I20260928 02:07:43.411381 16000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c"
format_stamp: "Formatted at 2026-09-28 02:07:43 on dist-test-slave-23bp"
server_key: "1cde41b6dbe852885c406b9a222d602c"
server_key_iv: "f15474c4753775cd2242df0521109fe1"
server_key_version: "encryptionkey@0"
I20260928 02:07:43.418808 16000 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260928 02:07:43.423848 16015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:43.425701 16000 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:07:43.425884 16000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c"
format_stamp: "Formatted at 2026-09-28 02:07:43 on dist-test-slave-23bp"
server_key: "1cde41b6dbe852885c406b9a222d602c"
server_key_iv: "f15474c4753775cd2242df0521109fe1"
server_key_version: "encryptionkey@0"
I20260928 02:07:43.426184 16000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:43.469513 16000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:43.473482 16000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:43.473771 16000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:43.474730 16000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:07:43.476354 16000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:07:43.476431 16000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:43.476605 16000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:07:43.476655 16000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:43.479600 15860 heartbeater.cc:503] Master 127.1.182.190:34479 was elected leader, sending a full tablet report...
I20260928 02:07:43.510149 16000 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:33905
I20260928 02:07:43.510241 16128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:33905 every 8 connection(s)
I20260928 02:07:43.511545 16000 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:07:43.514132 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 16000
I20260928 02:07:43.514359 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:07:43.515012 1754 external_mini_cluster.cc:1428] Setting key 36f46b9cf1c278a2766a41b008074a06
Sep 28 02:07:43 dist-test-slave-23bp krb5kdc[15624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561263, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:07:43.532968 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:43.514361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:39929 (local address 127.1.182.190:34479)
0928 02:07:43.514859 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:43.514866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:43.516087 (+ 1221us) server_negotiation.cc:407] Connection header received
0928 02:07:43.518691 (+ 2604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:43.518695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:43.518763 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:43.518887 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:43.520532 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.521321 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.522884 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.523355 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.523492 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:43.526901 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:43.526905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:43.526909 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:43.526943 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:43.529682 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:43.530577 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:43.530582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:43.530585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:43.530776 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:43.531299 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:43.531302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:43.531304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:43.531675 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:43.531789 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:43.532467 (+ 678us) server_negotiation.cc:299] Negotiation successful
0928 02:07:43.532707 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":241,"threads_started":1}
I20260928 02:07:43.533892 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:43.515017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.131:39929)
0928 02:07:43.515863 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:43.515930 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:43.518343 (+ 2413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:43.519234 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:43.519243 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:43.519771 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:43.520359 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:43.520377 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.521473 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.521478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:43.522730 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:43.522742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.523588 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.523595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:43.523833 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:43.524465 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:43.524492 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:43.526702 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:43.529835 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:43.529844 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:43.529861 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:43.530427 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:43.530895 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:43.530899 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:43.530902 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:43.531178 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:43.531782 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:43.531788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:43.532124 (+ 336us) client_negotiation.cc:769] Sending connection context
0928 02:07:43.532349 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 02:07:43.532761 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":302,"threads_started":1}
I20260928 02:07:43.535233 16129 heartbeater.cc:347] Connected to a master server at 127.1.182.190:34479
I20260928 02:07:43.535523 16129 heartbeater.cc:464] Registering TS with master...
I20260928 02:07:43.536211 16129 heartbeater.cc:511] Master 127.1.182.190:34479 requested a full tablet report, sending...
I20260928 02:07:43.537555 15670 ts_manager.cc:194] Registered new tserver with Master: 8374a3ebc2ec4b20ab72971cce4dcf1c (127.1.182.131:33905)
I20260928 02:07:43.538360 15670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:39929
I20260928 02:07:43.543599 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:07:43.561439 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:43.548183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49486 (local address 127.1.182.190:34479)
0928 02:07:43.548335 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:43.548339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:43.548788 (+ 449us) server_negotiation.cc:407] Connection header received
0928 02:07:43.549005 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:43.549009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:43.549074 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:43.549178 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:43.550568 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.551466 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.552844 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.553316 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.553453 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:43.555908 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:43.555913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:43.555916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:43.555947 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:43.558632 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:43.559458 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:43.559464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:43.559468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:43.559592 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:43.559969 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:43.559974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:43.559977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:43.560308 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:43.560564 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:43.560897 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 02:07:43.561124 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:07:43.565850 15670 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:07:43.572845 15670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:07:43.608065 16142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:43.584864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:33905 (local address 127.0.0.1:49770)
0928 02:07:43.597448 (+ 12584us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:43.597508 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:43.597792 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:43.600824 (+ 3032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:43.600829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:43.600853 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:07:43.601267 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:43.601281 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.603458 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.603466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:43.605304 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:43.605316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.607011 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.607017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:43.607232 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:43.607301 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 02:07:43.607423 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 02:07:43.607621 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12213,"mutex_wait_us":33,"thread_start_us":12026,"threads_started":1}
I20260928 02:07:43.609139 16145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:43.585567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49770 (local address 127.1.182.131:33905)
0928 02:07:43.600172 (+ 14605us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:43.600178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:43.600232 (+ 54us) server_negotiation.cc:407] Connection header received
0928 02:07:43.600341 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:43.600350 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:43.600614 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:43.601444 (+ 830us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:07:43.601739 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.603309 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.605472 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.606803 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.608323 (+ 1520us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:43.608538 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:43.608622 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 02:07:43.608802 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14120,"thread_start_us":14281,"threads_started":1}
I20260928 02:07:43.609259 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:43.598413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50704 (local address 127.1.182.130:45131)
0928 02:07:43.599338 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:43.599346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:43.600693 (+ 1347us) server_negotiation.cc:407] Connection header received
0928 02:07:43.600968 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:43.600978 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:43.601223 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:43.601418 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:07:43.602024 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.603505 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.606255 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.608028 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.608200 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:43.608413 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:43.608728 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 02:07:43.608904 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":662,"threads_started":1}
I20260928 02:07:43.609355 16147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:43.599412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:45131 (local address 127.0.0.1:50704)
0928 02:07:43.600035 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:43.600580 (+ 545us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:43.600799 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:43.601433 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:43.601440 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:43.601467 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:07:43.601846 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:43.601856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.603643 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.603653 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:43.606102 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:43.606112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.608280 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.608287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:43.608469 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:43.608566 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 02:07:43.608679 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 02:07:43.608861 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":79,"thread_start_us":319,"threads_started":1}
I20260928 02:07:43.609764 16144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:43.584994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.1.182.129:34723)
0928 02:07:43.598999 (+ 14005us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:43.599016 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:43.599042 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:07:43.599195 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:43.599204 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:43.599473 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:43.599691 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:07:43.600938 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.602441 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.606018 (+ 3577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.607533 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.607666 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:43.607896 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:43.608974 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0928 02:07:43.609149 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13820,"thread_start_us":13664,"threads_started":1}
I20260928 02:07:43.609874 16143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:43.584889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:34723 (local address 127.0.0.1:38354)
0928 02:07:43.598463 (+ 13574us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:43.598487 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:43.598611 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:43.600034 (+ 1423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:43.600045 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:43.600080 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:07:43.600562 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:43.600579 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.602573 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.602582 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:43.605844 (+ 3262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:43.605859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.608040 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.608051 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:43.608561 (+ 510us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:43.608594 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:07:43.608682 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 02:07:43.608841 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13426,"mutex_wait_us":7,"thread_start_us":13808,"threads_started":1}
I20260928 02:07:43.613217 16063 tablet_service.cc:1467] Processing CreateTablet for tablet 5044f88827c24d939218ee0662ad9229 (DEFAULT_TABLE table=test-table [id=c12fdf15a48a491cbff9c4109dfceb93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:07:43.613481 15930 tablet_service.cc:1467] Processing CreateTablet for tablet 5044f88827c24d939218ee0662ad9229 (DEFAULT_TABLE table=test-table [id=c12fdf15a48a491cbff9c4109dfceb93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:07:43.613916 15795 tablet_service.cc:1467] Processing CreateTablet for tablet 5044f88827c24d939218ee0662ad9229 (DEFAULT_TABLE table=test-table [id=c12fdf15a48a491cbff9c4109dfceb93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:07:43.614524 16063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5044f88827c24d939218ee0662ad9229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:07:43.614941 15930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5044f88827c24d939218ee0662ad9229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:07:43.615365 15795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5044f88827c24d939218ee0662ad9229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:07:43.625801 16148 tablet_bootstrap.cc:492] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c: Bootstrap starting.
I20260928 02:07:43.629081 16148 tablet_bootstrap.cc:654] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c: Neither blocks nor log segments found. Creating new log.
I20260928 02:07:43.630445 16148 log.cc:824] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:07:43.632277 16150 tablet_bootstrap.cc:492] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c: Bootstrap starting.
I20260928 02:07:43.631389 16149 tablet_bootstrap.cc:492] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea: Bootstrap starting.
I20260928 02:07:43.635257 16148 tablet_bootstrap.cc:492] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c: No bootstrap required, opened a new log
I20260928 02:07:43.635551 16148 ts_tablet_manager.cc:1397] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260928 02:07:43.636327 16150 tablet_bootstrap.cc:654] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c: Neither blocks nor log segments found. Creating new log.
I20260928 02:07:43.636327 16149 tablet_bootstrap.cc:654] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea: Neither blocks nor log segments found. Creating new log.
I20260928 02:07:43.637593 16150 log.cc:824] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:07:43.637593 16149 log.cc:824] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea: Log is configured to *not* fsync() on all Append() calls
I20260928 02:07:43.641633 16150 tablet_bootstrap.cc:492] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c: No bootstrap required, opened a new log
I20260928 02:07:43.641906 16150 ts_tablet_manager.cc:1397] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260928 02:07:43.642493 16149 tablet_bootstrap.cc:492] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea: No bootstrap required, opened a new log
I20260928 02:07:43.642753 16149 ts_tablet_manager.cc:1397] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.005s
I20260928 02:07:43.644726 16148 raft_consensus.cc:348] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } } peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } }
I20260928 02:07:43.645219 16148 raft_consensus.cc:374] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:07:43.645363 16148 raft_consensus.cc:729] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8374a3ebc2ec4b20ab72971cce4dcf1c, State: Initialized, Role: FOLLOWER
I20260928 02:07:43.646026 16148 consensus_queue.cc:260] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } } peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } }
I20260928 02:07:43.647419 16129 heartbeater.cc:503] Master 127.1.182.190:34479 was elected leader, sending a full tablet report...
I20260928 02:07:43.648135 16148 ts_tablet_manager.cc:1428] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260928 02:07:43.653213 16149 raft_consensus.cc:348] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } } peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } }
I20260928 02:07:43.653694 16149 raft_consensus.cc:374] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:07:43.653805 16149 raft_consensus.cc:729] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 814c38565bcd48eeb9de69fe0b86f6ea, State: Initialized, Role: FOLLOWER
I20260928 02:07:43.654317 16150 raft_consensus.cc:348] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } } peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } }
I20260928 02:07:43.654479 16149 consensus_queue.cc:260] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } } peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } }
I20260928 02:07:43.654702 16150 raft_consensus.cc:374] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:07:43.654809 16150 raft_consensus.cc:729] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305b82159a2e4591b5d7597a4620846c, State: Initialized, Role: FOLLOWER
I20260928 02:07:43.655509 16150 consensus_queue.cc:260] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } } peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } }
I20260928 02:07:43.656298 16149 ts_tablet_manager.cc:1428] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea: Time spent starting tablet: real 0.013s user 0.008s sys 0.007s
I20260928 02:07:43.656617 15995 heartbeater.cc:503] Master 127.1.182.190:34479 was elected leader, sending a full tablet report...
I20260928 02:07:43.657346 16150 ts_tablet_manager.cc:1428] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
W20260928 02:07:43.702298 15861 tablet.cc:2369] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:07:43.705541 15996 tablet.cc:2369] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:07:43.706892 16156 raft_consensus.cc:482] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:43.707188 16156 raft_consensus.cc:504] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } } peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } }
I20260928 02:07:43.708789 16156 leader_election.cc:288] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 814c38565bcd48eeb9de69fe0b86f6ea (127.1.182.129:34723), 8374a3ebc2ec4b20ab72971cce4dcf1c (127.1.182.131:33905)
I20260928 02:07:43.715466 16145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:43.709310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:40161 (local address 127.1.182.131:33905)
0928 02:07:43.709558 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:43.709566 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:43.710139 (+ 573us) server_negotiation.cc:407] Connection header received
0928 02:07:43.710313 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:43.710318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:43.710401 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:43.710481 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:07:43.711108 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.711868 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.713743 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.714481 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.714608 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:43.714660 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:43.715055 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 02:07:43.715209 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:07:43.715528 16144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:43.709193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:36117 (local address 127.1.182.129:34723)
0928 02:07:43.709433 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:43.709441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:43.710263 (+ 822us) server_negotiation.cc:407] Connection header received
0928 02:07:43.710431 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:43.710435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:43.710524 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:43.710626 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:07:43.711297 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.712054 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.713793 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.714611 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.714767 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:43.714838 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:43.715163 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 02:07:43.715308 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 02:07:43.715600 16157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:43.709113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:33905 (local address 127.1.182.130:40161)
0928 02:07:43.710027 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:43.710042 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:43.710170 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:43.710526 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:43.710532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:43.710557 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:07:43.710934 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:43.710943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.712001 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.712008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:43.713590 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:43.713603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.714701 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.714707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:43.714886 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:43.714913 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 02:07:43.715001 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 02:07:43.715160 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":23,"thread_start_us":527,"threads_started":1}
I20260928 02:07:43.715651 16158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:43.709197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:34723 (local address 127.1.182.130:36117)
0928 02:07:43.710136 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:43.710154 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:43.710308 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:43.710652 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:43.710658 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:43.710686 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:07:43.711135 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:43.711148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.712193 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.712199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:43.713615 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:43.713627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.714812 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.714817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:43.714942 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:43.714997 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 02:07:43.715123 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 02:07:43.715236 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":19,"thread_start_us":540,"threads_started":1}
I20260928 02:07:43.716583 16083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5044f88827c24d939218ee0662ad9229" candidate_uuid: "305b82159a2e4591b5d7597a4620846c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" is_pre_election: true
I20260928 02:07:43.716620 15815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5044f88827c24d939218ee0662ad9229" candidate_uuid: "305b82159a2e4591b5d7597a4620846c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" is_pre_election: true
I20260928 02:07:43.717067 16083 raft_consensus.cc:2454] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 305b82159a2e4591b5d7597a4620846c in term 0.
I20260928 02:07:43.717072 15815 raft_consensus.cc:2454] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 305b82159a2e4591b5d7597a4620846c in term 0.
I20260928 02:07:43.717715 15885 leader_election.cc:302] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 305b82159a2e4591b5d7597a4620846c, 814c38565bcd48eeb9de69fe0b86f6ea, 8374a3ebc2ec4b20ab72971cce4dcf1c; no voters:
I20260928 02:07:43.718122 16156 raft_consensus.cc:2781] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:07:43.718258 16156 raft_consensus.cc:482] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:07:43.718379 16156 raft_consensus.cc:3037] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:07:43.720600 16156 raft_consensus.cc:504] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } } peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } }
I20260928 02:07:43.721323 16156 leader_election.cc:288] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c [CANDIDATE]: Term 1 election: Requested vote from peers 814c38565bcd48eeb9de69fe0b86f6ea (127.1.182.129:34723), 8374a3ebc2ec4b20ab72971cce4dcf1c (127.1.182.131:33905)
I20260928 02:07:43.721673 16083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5044f88827c24d939218ee0662ad9229" candidate_uuid: "305b82159a2e4591b5d7597a4620846c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c"
I20260928 02:07:43.721791 15815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5044f88827c24d939218ee0662ad9229" candidate_uuid: "305b82159a2e4591b5d7597a4620846c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "814c38565bcd48eeb9de69fe0b86f6ea"
I20260928 02:07:43.721872 16083 raft_consensus.cc:3037] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:07:43.721985 15815 raft_consensus.cc:3037] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:07:43.724316 16083 raft_consensus.cc:2454] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 305b82159a2e4591b5d7597a4620846c in term 1.
I20260928 02:07:43.724376 15815 raft_consensus.cc:2454] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 305b82159a2e4591b5d7597a4620846c in term 1.
I20260928 02:07:43.724725 15882 leader_election.cc:302] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 305b82159a2e4591b5d7597a4620846c, 8374a3ebc2ec4b20ab72971cce4dcf1c; no voters:
I20260928 02:07:43.725025 16156 raft_consensus.cc:2781] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:07:43.725462 16156 raft_consensus.cc:686] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c [term 1 LEADER]: Becoming Leader. State: Replica: 305b82159a2e4591b5d7597a4620846c, State: Running, Role: LEADER
I20260928 02:07:43.725903 16156 consensus_queue.cc:237] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } } peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } }
I20260928 02:07:43.730449 15670 catalog_manager.cc:5495] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c reported cstate change: term changed from 0 to 1, leader changed from <none> to 305b82159a2e4591b5d7597a4620846c (127.1.182.130). New cstate: current_term: 1 leader_uuid: "305b82159a2e4591b5d7597a4620846c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } health_report { overall_health: HEALTHY } } }
W20260928 02:07:43.763243 16130 tablet.cc:2369] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:07:43.764472 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:43.759053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.1.182.130:45131)
0928 02:07:43.759266 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:43.759273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:43.759316 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:07:43.759583 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:43.759588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:43.759669 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:43.759820 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:07:43.760190 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.760922 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.762287 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.762692 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.762821 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:43.763375 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:07:43.763896 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:07:43.764059 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:43.764124 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 02:07:43.764203 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:07:43.772461 16083 raft_consensus.cc:1261] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 1 FOLLOWER]: Refusing update from remote peer 305b82159a2e4591b5d7597a4620846c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:07:43.772507 15815 raft_consensus.cc:1261] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Refusing update from remote peer 305b82159a2e4591b5d7597a4620846c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:07:43.773502 16159 consensus_queue.cc:1035] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c [LEADER]: Connected to new peer: Peer: permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:07:43.773873 16156 consensus_queue.cc:1035] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:07:43.786523 16163 mvcc.cc:205] Tried to move back new op lower bound from 7334138936402255872 to 7334138936224800768. Current Snapshot: MvccSnapshot[applied={T|T < 7334138936402255872}]
I20260928 02:07:43.799775 15670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:07:43.800043 15670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:07:43.803529 15670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223: Sending DeleteTablet for 3 replicas of tablet 5044f88827c24d939218ee0662ad9229
I20260928 02:07:43.804688 15795 tablet_service.cc:1514] Processing DeleteTablet for tablet 5044f88827c24d939218ee0662ad9229 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:07:43 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:38354
I20260928 02:07:43.804857 15930 tablet_service.cc:1514] Processing DeleteTablet for tablet 5044f88827c24d939218ee0662ad9229 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:07:43 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:50704
I20260928 02:07:43.805081 16063 tablet_service.cc:1514] Processing DeleteTablet for tablet 5044f88827c24d939218ee0662ad9229 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:07:43 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:49770
I20260928 02:07:43.805783 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 15731
I20260928 02:07:43.815860 16172 tablet_replica.cc:330] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea: stopping tablet replica
I20260928 02:07:43.816354 16172 raft_consensus.cc:2229] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:07:43.816896 16172 raft_consensus.cc:2258] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:07:43.824931 16172 ts_tablet_manager.cc:1905] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:07:43.835368 16172 ts_tablet_manager.cc:1918] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:07:43.835722 16172 log.cc:1196] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/wal/wals/5044f88827c24d939218ee0662ad9229
I20260928 02:07:43.836469 16172 ts_tablet_manager.cc:1939] T 5044f88827c24d939218ee0662ad9229 P 814c38565bcd48eeb9de69fe0b86f6ea: Deleting consensus metadata
I20260928 02:07:43.839304 15657 catalog_manager.cc:4841] TS 814c38565bcd48eeb9de69fe0b86f6ea (127.1.182.129:34723): tablet 5044f88827c24d939218ee0662ad9229 (table test-table [id=c12fdf15a48a491cbff9c4109dfceb93]) successfully deleted
I20260928 02:07:43.839627 16174 tablet_replica.cc:330] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c: stopping tablet replica
I20260928 02:07:43.840044 16174 raft_consensus.cc:2229] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:07:43.840509 16174 raft_consensus.cc:2258] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:07:43.841415 16173 tablet_replica.cc:330] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c: stopping tablet replica
I20260928 02:07:43.841841 16173 raft_consensus.cc:2229] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:07:43.842558 16173 raft_consensus.cc:2258] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:07:43.842722 16174 ts_tablet_manager.cc:1905] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 02:07:43 dist-test-slave-23bp krb5kdc[15624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:07:43.849526 16173 ts_tablet_manager.cc:1905] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:07:43.850062 16174 ts_tablet_manager.cc:1918] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:07:43.850260 16174 log.cc:1196] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/5044f88827c24d939218ee0662ad9229
I20260928 02:07:43.850821 16174 ts_tablet_manager.cc:1939] T 5044f88827c24d939218ee0662ad9229 P 8374a3ebc2ec4b20ab72971cce4dcf1c: Deleting consensus metadata
I20260928 02:07:43.852826 15654 catalog_manager.cc:4841] TS 8374a3ebc2ec4b20ab72971cce4dcf1c (127.1.182.131:33905): tablet 5044f88827c24d939218ee0662ad9229 (table test-table [id=c12fdf15a48a491cbff9c4109dfceb93]) successfully deleted
I20260928 02:07:43.858250 16144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:43.807227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38358 (local address 127.1.182.129:34723)
0928 02:07:43.807449 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:43.807457 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:43.823852 (+ 16395us) server_negotiation.cc:407] Connection header received
0928 02:07:43.831578 (+ 7726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:43.831587 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:43.831718 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:43.831814 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:43.840939 (+ 9125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.842054 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.844173 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.844782 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.844871 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:43.850659 (+ 5788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:43.850677 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:43.850695 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:43.850737 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:43.854527 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:43.855529 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:43.855538 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:43.855542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:43.855692 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:43.856205 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:43.856213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:43.856220 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:43.856659 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:43.856956 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:43.857602 (+ 646us) server_negotiation.cc:299] Negotiation successful
0928 02:07:43.857850 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:07:43.859786 15855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:07:43.860168 16173 ts_tablet_manager.cc:1918] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:07:43.860442 16173 log.cc:1196] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/5044f88827c24d939218ee0662ad9229
I20260928 02:07:43.861204 16173 ts_tablet_manager.cc:1939] T 5044f88827c24d939218ee0662ad9229 P 305b82159a2e4591b5d7597a4620846c: Deleting consensus metadata
I20260928 02:07:43.863497 15654 catalog_manager.cc:4841] TS 305b82159a2e4591b5d7597a4620846c (127.1.182.130:45131): tablet 5044f88827c24d939218ee0662ad9229 (table test-table [id=c12fdf15a48a491cbff9c4109dfceb93]) successfully deleted
I20260928 02:07:43.968231 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 15731
I20260928 02:07:43.988353 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 15866
Sep 28 02:07:43 dist-test-slave-23bp krb5kdc[15624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.130@KRBTEST.COM
I20260928 02:07:44.006299 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:43.989244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50710 (local address 127.1.182.130:45131)
0928 02:07:43.989425 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:43.989430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:43.989449 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:07:43.989628 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:43.989632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:43.989704 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:43.989786 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:43.991472 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.992331 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.994363 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:43.994797 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:43.994887 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:44.000079 (+ 5192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:44.000092 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:44.000117 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:44.000162 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:44.003376 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:44.004120 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:44.004128 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:44.004130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:44.004290 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:44.004671 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:44.004676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:44.004682 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:44.005048 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:44.005401 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:44.005692 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 02:07:44.005921 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:07:44.007047 15990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:07:44.122529 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 15866
I20260928 02:07:44.137727 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 16000
Sep 28 02:07:44 dist-test-slave-23bp krb5kdc[15624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.131@KRBTEST.COM
I20260928 02:07:44.152874 16145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:44.138419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49778 (local address 127.1.182.131:33905)
0928 02:07:44.138595 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:44.138602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:44.138691 (+ 89us) server_negotiation.cc:407] Connection header received
0928 02:07:44.138914 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:44.138918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:44.138979 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:44.139106 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:44.140398 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:44.141128 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:44.142443 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:44.142845 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:44.142967 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:44.147449 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:44.147463 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:44.147479 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:44.147515 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:44.150342 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:44.150940 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:44.150947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:44.150950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:44.151263 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:44.151600 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:44.151603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:44.151605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:44.151959 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:44.152173 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:44.152445 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 02:07:44.152615 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:07:44.153483 16124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:07:44.240929 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 16000
I20260928 02:07:44.258055 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 15638
I20260928 02:07:44.259259 15700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:07:44.368455 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 15638
I20260928 02:07:44.389848 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:34479
--webserver_interface=127.1.182.190
--webserver_port=45133
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:34479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:07:44.714946 16180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:44.715279 16180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:44.715456 16180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:44.722507 16180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:07:44.722609 16180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:44.722718 16180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:44.722820 16180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:07:44.722899 16180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:07:44.732527 16180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:34479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:34479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=45133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:44.733853 16180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:44.735225 16180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:44.758404 16186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:44.758526 16189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:44.758431 16187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:44.759182 16180 server_base.cc:1034] running on GCE node
I20260928 02:07:44.760260 16180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:44.761911 16180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:44.763171 16180 hybrid_clock.cc:648] HybridClock initialized: now 1790561264763105 us; error 60 us; skew 500 ppm
I20260928 02:07:44.763648 16180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:44 dist-test-slave-23bp krb5kdc[15624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561264, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:44.767508 16180 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:07:44.769217 16180 webserver.cc:460] Webserver started at http://127.1.182.190:45133/ using document root <none> and password file <none>
I20260928 02:07:44.770025 16180 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:44.770195 16180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:07:44.776559 16180 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:07:44.780041 16196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:44.781493 16180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:07:44.781697 16180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "317c6ee7d88947eb8b8ae4731321b223"
format_stamp: "Formatted at 2026-09-28 02:07:41 on dist-test-slave-23bp"
server_key: "b63465a1e8561cce45bee503641a6be1"
server_key_iv: "fffdc50cad54da34f43e612c993ab124"
server_key_version: "encryptionkey@0"
I20260928 02:07:44.782830 16180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:44.810983 16180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:44.815287 16180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:44.815579 16180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:44.834975 16180 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:34479
I20260928 02:07:44.834977 16247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:34479 every 8 connection(s)
I20260928 02:07:44.836597 16180 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:07:44.842270 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 16180
I20260928 02:07:44.843133 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:34723
--local_ip_for_outbound_sockets=127.1.182.129
--tserver_master_addrs=127.1.182.190:34479
--webserver_port=32799
--webserver_interface=127.1.182.129
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:07:44.843734 16248 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:07:44.849166 16248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223: Bootstrap starting.
I20260928 02:07:44.877291 16248 log.cc:824] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223: Log is configured to *not* fsync() on all Append() calls
I20260928 02:07:44.892861 16248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:07:44.893530 16248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223: Bootstrap complete.
I20260928 02:07:44.904654 16248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317c6ee7d88947eb8b8ae4731321b223" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 34479 } }
I20260928 02:07:44.905552 16248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 317c6ee7d88947eb8b8ae4731321b223, State: Initialized, Role: FOLLOWER
I20260928 02:07:44.906194 16248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317c6ee7d88947eb8b8ae4731321b223" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 34479 } }
I20260928 02:07:44.906454 16248 raft_consensus.cc:388] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:07:44.906586 16248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:07:44.906785 16248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:07:44.910778 16248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317c6ee7d88947eb8b8ae4731321b223" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 34479 } }
I20260928 02:07:44.911391 16248 leader_election.cc:302] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 317c6ee7d88947eb8b8ae4731321b223; no voters:
I20260928 02:07:44.911983 16248 leader_election.cc:288] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:07:44.912187 16252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:07:44.913533 16252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 2 LEADER]: Becoming Leader. State: Replica: 317c6ee7d88947eb8b8ae4731321b223, State: Running, Role: LEADER
I20260928 02:07:44.914264 16248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:07:44.914176 16252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317c6ee7d88947eb8b8ae4731321b223" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 34479 } }
I20260928 02:07:44.917583 16254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 317c6ee7d88947eb8b8ae4731321b223. Latest consensus state: current_term: 2 leader_uuid: "317c6ee7d88947eb8b8ae4731321b223" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317c6ee7d88947eb8b8ae4731321b223" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 34479 } } }
I20260928 02:07:44.918051 16254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [sys.catalog]: This master's current role is: LEADER
I20260928 02:07:44.922237 16253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "317c6ee7d88947eb8b8ae4731321b223" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317c6ee7d88947eb8b8ae4731321b223" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 34479 } } }
I20260928 02:07:44.922631 16253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [sys.catalog]: This master's current role is: LEADER
I20260928 02:07:44.929821 16261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:07:44.938606 16261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:07:44.940136 16261 catalog_manager.cc:1259] Loaded cluster ID: fe7d6cb0497a4cd7aeee849a06f00b76
I20260928 02:07:44.940305 16261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:07:44.943493 16261 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 02:07:44.961683 16261 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 02:07:44.971195 16261 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:07:44.973078 16261 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223: Loaded TSK: 0
I20260928 02:07:44.974617 16261 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:07:45.214295 16250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:45.214567 16250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:45.214712 16250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:45.222101 16250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:45.222208 16250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:45.222378 16250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:07:45.232316 16250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:34723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=32799
--tserver_master_addrs=127.1.182.190:34479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:45.233635 16250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:45.235003 16250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:45.249145 16277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:45.249066 16279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:45.249140 16276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:45.250849 16250 server_base.cc:1034] running on GCE node
I20260928 02:07:45.251528 16250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:45.252575 16250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:45.253840 16250 hybrid_clock.cc:648] HybridClock initialized: now 1790561265253777 us; error 59 us; skew 500 ppm
I20260928 02:07:45.254180 16250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:45 dist-test-slave-23bp krb5kdc[15624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561265, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:45.262974 16250 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:07:45.276859 16250 webserver.cc:460] Webserver started at http://127.1.182.129:32799/ using document root <none> and password file <none>
I20260928 02:07:45.278079 16250 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:45.278295 16250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:07:45.286181 16250 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:07:45.289501 16286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:45.291168 16250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:07:45.291352 16250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "814c38565bcd48eeb9de69fe0b86f6ea"
format_stamp: "Formatted at 2026-09-28 02:07:42 on dist-test-slave-23bp"
server_key: "c259bc7e052e007f91ada4900e261488"
server_key_iv: "44c684cfbf17e6d7767c0646e6210fb3"
server_key_version: "encryptionkey@0"
I20260928 02:07:45.292428 16250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:45.309719 16250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:45.314736 16250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:45.315116 16250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:45.316244 16250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:07:45.317685 16250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:07:45.317812 16250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:45.317962 16250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:07:45.318078 16250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:45.352634 16250 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:34723
I20260928 02:07:45.352741 16399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:34723 every 8 connection(s)
I20260928 02:07:45.354194 16250 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:07:45.356376 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 16250
I20260928 02:07:45.357136 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:45131
--local_ip_for_outbound_sockets=127.1.182.130
--tserver_master_addrs=127.1.182.190:34479
--webserver_port=46751
--webserver_interface=127.1.182.130
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:07:45 dist-test-slave-23bp krb5kdc[15624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561265, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:07:45.381299 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:45.357333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:51113 (local address 127.1.182.190:34479)
0928 02:07:45.357980 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:45.357999 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:45.358662 (+ 663us) server_negotiation.cc:407] Connection header received
0928 02:07:45.360995 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:45.361018 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:45.361548 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:45.361995 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:45.363632 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:45.365449 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:45.368260 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:45.368926 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:45.369090 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:45.373554 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:45.373562 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:45.373576 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:45.373615 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:45.376679 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:45.377433 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:45.377440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:45.377446 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:45.377605 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:45.378044 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:45.378048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:45.378050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:45.378540 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:45.378766 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:45.379587 (+ 821us) server_negotiation.cc:299] Negotiation successful
0928 02:07:45.379957 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":253,"threads_started":1}
I20260928 02:07:45.381300 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:45.357591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:51113)
0928 02:07:45.358466 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:45.358513 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:45.360321 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:45.362125 (+ 1804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:45.362133 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:45.362714 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:45.363348 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:45.363368 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:45.365643 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:45.365657 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:45.368073 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:45.368101 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:45.369213 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:45.369223 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:45.369573 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:45.370430 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:45.370473 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:45.373266 (+ 2793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:45.376830 (+ 3564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:45.376840 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:45.376857 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:45.377279 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:45.377723 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:45.377729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:45.377732 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:45.377916 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:45.378682 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:45.378690 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:45.379039 (+ 349us) client_negotiation.cc:769] Sending connection context
0928 02:07:45.379371 (+ 332us) client_negotiation.cc:240] Negotiation successful
0928 02:07:45.379945 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":383,"threads_started":1}
I20260928 02:07:45.384728 16400 heartbeater.cc:347] Connected to a master server at 127.1.182.190:34479
I20260928 02:07:45.385046 16400 heartbeater.cc:464] Registering TS with master...
I20260928 02:07:45.385936 16400 heartbeater.cc:511] Master 127.1.182.190:34479 requested a full tablet report, sending...
I20260928 02:07:45.388249 16213 ts_manager.cc:194] Registered new tserver with Master: 814c38565bcd48eeb9de69fe0b86f6ea (127.1.182.129:34723)
I20260928 02:07:45.389701 16213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:51113
W20260928 02:07:45.700969 16404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:45.701400 16404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:45.701586 16404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:45.708875 16404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:45.709033 16404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:45.709218 16404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:07:45.719172 16404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:45131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=46751
--tserver_master_addrs=127.1.182.190:34479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:45.720494 16404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:45.721913 16404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:45.735589 16410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:45.735651 16409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:45.735675 16412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:45.736402 16404 server_base.cc:1034] running on GCE node
I20260928 02:07:45.736944 16404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:45.737884 16404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:45.739176 16404 hybrid_clock.cc:648] HybridClock initialized: now 1790561265739090 us; error 54 us; skew 500 ppm
I20260928 02:07:45.739465 16404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:45 dist-test-slave-23bp krb5kdc[15624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561265, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:45.742823 16404 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:07:45.744321 16404 webserver.cc:460] Webserver started at http://127.1.182.130:46751/ using document root <none> and password file <none>
I20260928 02:07:45.745105 16404 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:45.745244 16404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:07:45.750494 16404 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:07:45.753238 16419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:45.754627 16404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:07:45.754807 16404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "305b82159a2e4591b5d7597a4620846c"
format_stamp: "Formatted at 2026-09-28 02:07:42 on dist-test-slave-23bp"
server_key: "8ecac998a08a48ea267e1260040402e1"
server_key_iv: "149eee9f057f5584d28d400f2c307e3d"
server_key_version: "encryptionkey@0"
I20260928 02:07:45.755928 16404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:45.776762 16404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:45.780817 16404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:45.781073 16404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:45.782056 16404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:07:45.783425 16404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:07:45.783520 16404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:45.783670 16404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:07:45.783761 16404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:45.817086 16404 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:45131
I20260928 02:07:45.817178 16533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:45131 every 8 connection(s)
I20260928 02:07:45.818562 16404 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:07:45.824646 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 16404
I20260928 02:07:45.825417 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:33905
--local_ip_for_outbound_sockets=127.1.182.131
--tserver_master_addrs=127.1.182.190:34479
--webserver_port=37413
--webserver_interface=127.1.182.131
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:07:45 dist-test-slave-23bp krb5kdc[15624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561265, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:07:45.842245 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:45.821158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:39263 (local address 127.1.182.190:34479)
0928 02:07:45.821326 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:45.821331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:45.822547 (+ 1216us) server_negotiation.cc:407] Connection header received
0928 02:07:45.824394 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:45.824399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:45.824467 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:45.824546 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:45.826671 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:45.827507 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:45.830043 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:45.830526 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:45.830649 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:45.835164 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:45.835171 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:45.835176 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:45.835215 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:45.838587 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:45.839572 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:45.839578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:45.839582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:45.839728 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:45.840248 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:45.840253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:45.840257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:45.840622 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:45.840758 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:45.841668 (+ 910us) server_negotiation.cc:299] Negotiation successful
0928 02:07:45.841898 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:07:45.843616 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:45.821692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.130:39263)
0928 02:07:45.822368 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:45.822404 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:45.824052 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:45.824857 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:45.824869 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:45.825643 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:45.826483 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:45.826510 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:45.827681 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:45.827689 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:45.829882 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:45.829901 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:45.830784 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:45.830794 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:45.831167 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:45.831918 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:45.831957 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:45.834889 (+ 2932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:45.838750 (+ 3861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:45.838762 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:45.838781 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:45.839437 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:45.839853 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:45.839861 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:45.839868 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:45.840125 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:45.840762 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:45.840773 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:45.841230 (+ 457us) client_negotiation.cc:769] Sending connection context
0928 02:07:45.841535 (+ 305us) client_negotiation.cc:240] Negotiation successful
0928 02:07:45.842120 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":262,"threads_started":1}
I20260928 02:07:45.845238 16534 heartbeater.cc:347] Connected to a master server at 127.1.182.190:34479
I20260928 02:07:45.845592 16534 heartbeater.cc:464] Registering TS with master...
I20260928 02:07:45.846565 16534 heartbeater.cc:511] Master 127.1.182.190:34479 requested a full tablet report, sending...
I20260928 02:07:45.848330 16213 ts_manager.cc:194] Registered new tserver with Master: 305b82159a2e4591b5d7597a4620846c (127.1.182.130:45131)
I20260928 02:07:45.849438 16213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:39263
W20260928 02:07:46.160422 16537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:46.160738 16537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:46.160907 16537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:46.168097 16537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:46.168246 16537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:46.168431 16537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:07:46.177896 16537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:33905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=37413
--tserver_master_addrs=127.1.182.190:34479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:46.179175 16537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:46.180474 16537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:46.193420 16542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:46.193391 16543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:46.193475 16545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:46.194551 16537 server_base.cc:1034] running on GCE node
I20260928 02:07:46.195181 16537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:46.196107 16537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:46.197293 16537 hybrid_clock.cc:648] HybridClock initialized: now 1790561266197242 us; error 53 us; skew 500 ppm
I20260928 02:07:46.197606 16537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:46 dist-test-slave-23bp krb5kdc[15624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561266, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:46.200753 16537 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:07:46.202129 16537 webserver.cc:460] Webserver started at http://127.1.182.131:37413/ using document root <none> and password file <none>
I20260928 02:07:46.202811 16537 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:46.202958 16537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:07:46.208160 16537 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:07:46.210842 16552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:46.212251 16537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:07:46.212428 16537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c"
format_stamp: "Formatted at 2026-09-28 02:07:43 on dist-test-slave-23bp"
server_key: "1cde41b6dbe852885c406b9a222d602c"
server_key_iv: "f15474c4753775cd2242df0521109fe1"
server_key_version: "encryptionkey@0"
I20260928 02:07:46.213392 16537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:46.234558 16537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:46.238479 16537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:46.238763 16537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:46.239799 16537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:07:46.241110 16537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:07:46.241273 16537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:46.241428 16537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:07:46.241523 16537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:46.274370 16537 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:33905
I20260928 02:07:46.274456 16664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:33905 every 8 connection(s)
I20260928 02:07:46.275909 16537 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:07:46.284669 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 16537
Sep 28 02:07:46 dist-test-slave-23bp krb5kdc[15624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561266, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:07:46.297195 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.278450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:43885 (local address 127.1.182.190:34479)
0928 02:07:46.278661 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:46.278666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:46.279864 (+ 1198us) server_negotiation.cc:407] Connection header received
0928 02:07:46.281729 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:46.281735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:46.281826 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:46.281934 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:46.283459 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.284185 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.286309 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.287031 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.287187 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.291032 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:46.291037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:46.291040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:46.291113 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:46.293960 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:46.294675 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:46.294680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:46.294690 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:46.294834 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:46.295354 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:46.295359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:46.295362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:46.295663 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:46.295859 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:46.296629 (+ 770us) server_negotiation.cc:299] Negotiation successful
0928 02:07:46.296859 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 02:07:46.298417 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.278969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.131:43885)
0928 02:07:46.279708 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:46.279751 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:46.281446 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:46.282174 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:46.282183 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:46.282670 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:46.283296 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:46.283314 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.284339 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.284349 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:46.286138 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:46.286152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.287288 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.287297 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:46.287645 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.288410 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:46.288446 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:46.290805 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:46.294109 (+ 3304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:46.294117 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:46.294132 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:46.294537 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:46.294952 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:46.294969 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:46.294972 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:46.295237 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:46.295833 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:46.295842 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:46.296185 (+ 343us) client_negotiation.cc:769] Sending connection context
0928 02:07:46.296521 (+ 336us) client_negotiation.cc:240] Negotiation successful
0928 02:07:46.297026 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":265,"threads_started":1}
I20260928 02:07:46.300055 16665 heartbeater.cc:347] Connected to a master server at 127.1.182.190:34479
I20260928 02:07:46.300209 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.285593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49496 (local address 127.1.182.190:34479)
0928 02:07:46.286089 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:46.286094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:46.286454 (+ 360us) server_negotiation.cc:407] Connection header received
0928 02:07:46.286627 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:46.286630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:46.286701 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:46.286842 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:46.288894 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.289703 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.291442 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.291906 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.292042 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.294441 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:46.294445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:46.294448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:46.294477 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:46.297367 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:46.298202 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:46.298206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:46.298208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:46.298367 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:46.298834 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:46.298840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:46.298845 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:46.299177 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:46.299426 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:46.299699 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 02:07:46.299858 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":242,"threads_started":1}
I20260928 02:07:46.300402 16665 heartbeater.cc:464] Registering TS with master...
I20260928 02:07:46.301182 16665 heartbeater.cc:511] Master 127.1.182.190:34479 requested a full tablet report, sending...
I20260928 02:07:46.302709 16212 ts_manager.cc:194] Registered new tserver with Master: 8374a3ebc2ec4b20ab72971cce4dcf1c (127.1.182.131:33905)
I20260928 02:07:46.303660 16212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:43885
I20260928 02:07:46.314797 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:07:46.321756 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.315748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49512 (local address 127.1.182.190:34479)
0928 02:07:46.315927 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:46.315931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:46.316393 (+ 462us) server_negotiation.cc:407] Connection header received
0928 02:07:46.316651 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:46.316655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:46.316792 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:46.316941 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:07:46.317371 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.318069 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.319550 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.319966 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.320096 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.320703 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:07:46.321200 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:07:46.321362 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:46.321419 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 02:07:46.321496 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:07:46.322585 16212 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:07:46.329751 16212 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:07:46.353408 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.340946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:45131 (local address 127.0.0.1:50724)
0928 02:07:46.342006 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:46.342060 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:46.342237 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:46.344050 (+ 1813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:46.344058 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:46.344088 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:07:46.344652 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:46.344666 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.347394 (+ 2728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.347401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:46.350331 (+ 2930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:46.350347 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.352390 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.352396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:46.352579 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.352628 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 02:07:46.352733 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:07:46.352948 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":28,"thread_start_us":2271,"threads_started":1}
I20260928 02:07:46.353844 16678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.341717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38366 (local address 127.1.182.129:34723)
0928 02:07:46.343001 (+ 1284us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:46.343012 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:46.343626 (+ 614us) server_negotiation.cc:407] Connection header received
0928 02:07:46.343917 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:46.343928 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:46.344195 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:46.344372 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:07:46.345364 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.347193 (+ 1829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.350674 (+ 3481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.352218 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.352379 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.352604 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:46.353236 (+ 632us) server_negotiation.cc:299] Negotiation successful
0928 02:07:46.353440 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1086,"thread_start_us":980,"threads_started":1}
I20260928 02:07:46.354645 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.341138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:34723 (local address 127.0.0.1:38366)
0928 02:07:46.343515 (+ 2377us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:46.343534 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:46.343724 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:46.344705 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:46.344716 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:46.344746 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:07:46.345169 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:46.345183 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.347408 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.347412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:46.350520 (+ 3108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:46.350539 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.352785 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.352792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:46.352994 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.353027 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:07:46.354126 (+ 1099us) client_negotiation.cc:240] Negotiation successful
0928 02:07:46.354272 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2232,"thread_start_us":793,"threads_started":1}
I20260928 02:07:46.355578 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.341188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50724 (local address 127.1.182.130:45131)
0928 02:07:46.343363 (+ 2175us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:46.343372 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:46.343408 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:07:46.343534 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:46.343546 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:46.343814 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:46.345234 (+ 1420us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:07:46.345464 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.347222 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.350510 (+ 3288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.352059 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.354293 (+ 2234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.354490 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:46.354563 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 02:07:46.355271 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1991,"thread_start_us":1802,"threads_started":1}
I20260928 02:07:46.356024 16679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.343609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:33905 (local address 127.0.0.1:49790)
0928 02:07:46.348314 (+ 4705us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:46.348334 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:46.348457 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:46.349149 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:46.349153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:46.349177 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:07:46.349615 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:46.349623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.351217 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.351223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:46.353274 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:46.353288 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.355232 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.355240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:46.355407 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.355440 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:07:46.355519 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:07:46.355690 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4539,"thread_start_us":4390,"threads_started":1}
I20260928 02:07:46.357576 16680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.343906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.1.182.131:33905)
0928 02:07:46.347946 (+ 4040us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:46.347955 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:46.348578 (+ 623us) server_negotiation.cc:407] Connection header received
0928 02:07:46.348716 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:46.348725 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:46.348944 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:46.349165 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:07:46.349796 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.351037 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.353499 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.354866 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.355014 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.356362 (+ 1348us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:46.356429 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 02:07:46.356589 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3859,"thread_start_us":3696,"threads_started":1}
I20260928 02:07:46.358870 16334 tablet_service.cc:1467] Processing CreateTablet for tablet 850dccccfea74309b5d291f384383667 (DEFAULT_TABLE table=test-table [id=8a61308fa1034f8cafa738832d026c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:07:46.360421 16334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 850dccccfea74309b5d291f384383667. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:07:46.360090 16467 tablet_service.cc:1467] Processing CreateTablet for tablet 850dccccfea74309b5d291f384383667 (DEFAULT_TABLE table=test-table [id=8a61308fa1034f8cafa738832d026c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:07:46.360858 16600 tablet_service.cc:1467] Processing CreateTablet for tablet 850dccccfea74309b5d291f384383667 (DEFAULT_TABLE table=test-table [id=8a61308fa1034f8cafa738832d026c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:07:46.361408 16467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 850dccccfea74309b5d291f384383667. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:07:46.362169 16600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 850dccccfea74309b5d291f384383667. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:07:46.374388 16681 tablet_bootstrap.cc:492] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea: Bootstrap starting.
I20260928 02:07:46.376451 16682 tablet_bootstrap.cc:492] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c: Bootstrap starting.
I20260928 02:07:46.378012 16683 tablet_bootstrap.cc:492] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c: Bootstrap starting.
I20260928 02:07:46.379163 16681 tablet_bootstrap.cc:654] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea: Neither blocks nor log segments found. Creating new log.
I20260928 02:07:46.380147 16681 log.cc:824] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea: Log is configured to *not* fsync() on all Append() calls
I20260928 02:07:46.380256 16682 tablet_bootstrap.cc:654] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c: Neither blocks nor log segments found. Creating new log.
I20260928 02:07:46.381560 16682 log.cc:824] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:07:46.381635 16683 tablet_bootstrap.cc:654] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c: Neither blocks nor log segments found. Creating new log.
I20260928 02:07:46.382782 16683 log.cc:824] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:07:46.383045 16681 tablet_bootstrap.cc:492] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea: No bootstrap required, opened a new log
I20260928 02:07:46.383419 16681 ts_tablet_manager.cc:1397] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260928 02:07:46.385162 16682 tablet_bootstrap.cc:492] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c: No bootstrap required, opened a new log
I20260928 02:07:46.385644 16682 ts_tablet_manager.cc:1397] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260928 02:07:46.391559 16681 raft_consensus.cc:348] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } } peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } }
I20260928 02:07:46.392047 16681 raft_consensus.cc:374] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:07:46.392223 16681 raft_consensus.cc:729] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 814c38565bcd48eeb9de69fe0b86f6ea, State: Initialized, Role: FOLLOWER
I20260928 02:07:46.392561 16400 heartbeater.cc:503] Master 127.1.182.190:34479 was elected leader, sending a full tablet report...
I20260928 02:07:46.392976 16681 consensus_queue.cc:260] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } } peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } }
I20260928 02:07:46.395462 16683 tablet_bootstrap.cc:492] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c: No bootstrap required, opened a new log
I20260928 02:07:46.395936 16683 ts_tablet_manager.cc:1397] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s
I20260928 02:07:46.397312 16682 raft_consensus.cc:348] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } } peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } }
I20260928 02:07:46.397842 16682 raft_consensus.cc:374] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:07:46.397987 16682 raft_consensus.cc:729] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305b82159a2e4591b5d7597a4620846c, State: Initialized, Role: FOLLOWER
I20260928 02:07:46.398792 16681 ts_tablet_manager.cc:1428] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
I20260928 02:07:46.398824 16682 consensus_queue.cc:260] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } } peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } }
I20260928 02:07:46.400053 16534 heartbeater.cc:503] Master 127.1.182.190:34479 was elected leader, sending a full tablet report...
I20260928 02:07:46.402033 16682 ts_tablet_manager.cc:1428] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c: Time spent starting tablet: real 0.016s user 0.006s sys 0.010s
I20260928 02:07:46.405979 16683 raft_consensus.cc:348] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } } peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } }
I20260928 02:07:46.406365 16683 raft_consensus.cc:374] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:07:46.406486 16683 raft_consensus.cc:729] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8374a3ebc2ec4b20ab72971cce4dcf1c, State: Initialized, Role: FOLLOWER
I20260928 02:07:46.407025 16683 consensus_queue.cc:260] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } } peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } }
I20260928 02:07:46.408034 16665 heartbeater.cc:503] Master 127.1.182.190:34479 was elected leader, sending a full tablet report...
I20260928 02:07:46.408663 16683 ts_tablet_manager.cc:1428] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260928 02:07:46.455590 16689 raft_consensus.cc:482] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:46.455899 16689 raft_consensus.cc:504] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } } peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } }
I20260928 02:07:46.456504 16687 raft_consensus.cc:482] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:46.456780 16687 raft_consensus.cc:504] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } } peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } }
I20260928 02:07:46.457799 16689 leader_election.cc:288] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 305b82159a2e4591b5d7597a4620846c (127.1.182.130:45131), 814c38565bcd48eeb9de69fe0b86f6ea (127.1.182.129:34723)
I20260928 02:07:46.458490 16687 leader_election.cc:288] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 305b82159a2e4591b5d7597a4620846c (127.1.182.130:45131), 8374a3ebc2ec4b20ab72971cce4dcf1c (127.1.182.131:33905)
I20260928 02:07:46.465808 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.458162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:45131 (local address 127.1.182.131:59195)
0928 02:07:46.458678 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:46.458698 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:46.458812 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:46.459435 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:46.459440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:46.459460 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:07:46.459659 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:46.459670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.461128 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.461133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:46.463186 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:46.463204 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.465021 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.465032 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:46.465248 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.465281 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:07:46.465370 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 02:07:46.465498 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":23}
I20260928 02:07:46.466835 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.458727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:59195 (local address 127.1.182.130:45131)
0928 02:07:46.458913 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:46.458922 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:46.458946 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:07:46.459029 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:46.459036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:46.459164 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:46.459260 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:07:46.459825 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.460946 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.463384 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.464681 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.464848 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.464938 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:46.466395 (+ 1457us) server_negotiation.cc:299] Negotiation successful
0928 02:07:46.466533 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:07:46.467511 16678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.458399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:54389 (local address 127.1.182.129:34723)
0928 02:07:46.459085 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:46.459099 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:46.459357 (+ 258us) server_negotiation.cc:407] Connection header received
0928 02:07:46.459635 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:46.459640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:46.459714 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:46.459776 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:07:46.460725 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.461863 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.465050 (+ 3187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.466371 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.466493 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.466600 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:46.467021 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 02:07:46.467218 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":538}
I20260928 02:07:46.467893 16487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850dccccfea74309b5d291f384383667" candidate_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305b82159a2e4591b5d7597a4620846c" is_pre_election: true
I20260928 02:07:46.468259 16690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.458268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:34723 (local address 127.1.182.131:54389)
0928 02:07:46.459255 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:46.459272 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:46.459453 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:46.460166 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:46.460171 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:46.460198 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:07:46.460544 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:46.460554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.462053 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.462057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:46.464873 (+ 2816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:46.464891 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.466657 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.466662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:46.466836 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.466868 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:07:46.467727 (+ 859us) client_negotiation.cc:240] Negotiation successful
0928 02:07:46.467868 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":78,"thread_start_us":661,"threads_started":1}
I20260928 02:07:46.468448 16487 raft_consensus.cc:2454] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8374a3ebc2ec4b20ab72971cce4dcf1c in term 0.
I20260928 02:07:46.468703 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.458742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:45131 (local address 127.1.182.129:41683)
0928 02:07:46.459615 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:46.459637 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:46.459753 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:46.460989 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:46.460997 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:46.461035 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:07:46.461645 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:46.461662 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.463720 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.463726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:46.465932 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:46.465946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.467793 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.467799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:46.467994 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.468042 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 02:07:46.468115 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:07:46.468266 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":1,"thread_start_us":725,"threads_started":1}
I20260928 02:07:46.469523 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.459311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:41683 (local address 127.1.182.130:45131)
0928 02:07:46.460527 (+ 1216us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:46.460535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:46.460563 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:07:46.460696 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:46.460704 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:46.460792 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:46.461816 (+ 1024us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:07:46.462147 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.463527 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.466112 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.467536 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.468960 (+ 1424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.469029 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:46.469082 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 02:07:46.469212 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1039,"thread_start_us":909,"threads_started":1}
I20260928 02:07:46.469336 16354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850dccccfea74309b5d291f384383667" candidate_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" is_pre_election: true
I20260928 02:07:46.469753 16354 raft_consensus.cc:2454] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8374a3ebc2ec4b20ab72971cce4dcf1c in term 0.
I20260928 02:07:46.469775 16553 leader_election.cc:302] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 305b82159a2e4591b5d7597a4620846c, 8374a3ebc2ec4b20ab72971cce4dcf1c; no voters:
I20260928 02:07:46.470304 16487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850dccccfea74309b5d291f384383667" candidate_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305b82159a2e4591b5d7597a4620846c" is_pre_election: true
I20260928 02:07:46.470335 16689 raft_consensus.cc:2781] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:07:46.470497 16689 raft_consensus.cc:482] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:07:46.470499 16487 raft_consensus.cc:2454] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 814c38565bcd48eeb9de69fe0b86f6ea in term 0.
I20260928 02:07:46.470616 16689 raft_consensus.cc:3037] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:07:46.470925 16680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.460048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:49213 (local address 127.1.182.131:33905)
0928 02:07:46.460227 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:46.460234 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:46.463614 (+ 3380us) server_negotiation.cc:407] Connection header received
0928 02:07:46.463786 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:46.463791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:46.463857 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:46.463909 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:07:46.464717 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.465844 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.468447 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.469669 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.469799 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.469862 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:46.470526 (+ 664us) server_negotiation.cc:299] Negotiation successful
0928 02:07:46.470670 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":46}
I20260928 02:07:46.470892 16287 leader_election.cc:302] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 305b82159a2e4591b5d7597a4620846c, 814c38565bcd48eeb9de69fe0b86f6ea; no voters:
I20260928 02:07:46.471282 16687 raft_consensus.cc:2781] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:07:46.471439 16687 raft_consensus.cc:482] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:07:46.471581 16687 raft_consensus.cc:3037] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:07:46.471681 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.459875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:33905 (local address 127.1.182.129:49213)
0928 02:07:46.463497 (+ 3622us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:46.463520 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:46.463636 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:46.464074 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:46.464078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:46.464115 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:07:46.464528 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:46.464539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.466053 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.466060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:46.468265 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:46.468278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.470147 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.470154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:46.470321 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.470364 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:07:46.471168 (+ 804us) client_negotiation.cc:240] Negotiation successful
0928 02:07:46.471310 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3508,"thread_start_us":3307,"threads_started":1}
I20260928 02:07:46.472553 16620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850dccccfea74309b5d291f384383667" candidate_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" is_pre_election: true
I20260928 02:07:46.473495 16689 raft_consensus.cc:504] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } } peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } }
I20260928 02:07:46.473963 16620 raft_consensus.cc:2379] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 814c38565bcd48eeb9de69fe0b86f6ea in current term 1: Already voted for candidate 8374a3ebc2ec4b20ab72971cce4dcf1c in this term.
I20260928 02:07:46.474165 16689 leader_election.cc:288] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [CANDIDATE]: Term 1 election: Requested vote from peers 305b82159a2e4591b5d7597a4620846c (127.1.182.130:45131), 814c38565bcd48eeb9de69fe0b86f6ea (127.1.182.129:34723)
I20260928 02:07:46.474517 16487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850dccccfea74309b5d291f384383667" candidate_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305b82159a2e4591b5d7597a4620846c"
I20260928 02:07:46.474462 16687 raft_consensus.cc:504] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } } peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } }
I20260928 02:07:46.474713 16487 raft_consensus.cc:3037] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:07:46.474841 16354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850dccccfea74309b5d291f384383667" candidate_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "814c38565bcd48eeb9de69fe0b86f6ea"
I20260928 02:07:46.475136 16354 raft_consensus.cc:2379] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8374a3ebc2ec4b20ab72971cce4dcf1c in current term 1: Already voted for candidate 814c38565bcd48eeb9de69fe0b86f6ea in this term.
I20260928 02:07:46.475229 16687 leader_election.cc:288] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 1 election: Requested vote from peers 305b82159a2e4591b5d7597a4620846c (127.1.182.130:45131), 8374a3ebc2ec4b20ab72971cce4dcf1c (127.1.182.131:33905)
I20260928 02:07:46.475880 16486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850dccccfea74309b5d291f384383667" candidate_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305b82159a2e4591b5d7597a4620846c"
I20260928 02:07:46.476244 16620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850dccccfea74309b5d291f384383667" candidate_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c"
I20260928 02:07:46.476459 16620 raft_consensus.cc:2379] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 814c38565bcd48eeb9de69fe0b86f6ea in current term 1: Already voted for candidate 8374a3ebc2ec4b20ab72971cce4dcf1c in this term.
I20260928 02:07:46.476833 16287 leader_election.cc:302] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 814c38565bcd48eeb9de69fe0b86f6ea; no voters: 305b82159a2e4591b5d7597a4620846c, 8374a3ebc2ec4b20ab72971cce4dcf1c
I20260928 02:07:46.477124 16687 raft_consensus.cc:2726] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 02:07:46.477777 16487 raft_consensus.cc:2454] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8374a3ebc2ec4b20ab72971cce4dcf1c in term 1.
I20260928 02:07:46.478189 16553 leader_election.cc:302] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 305b82159a2e4591b5d7597a4620846c, 8374a3ebc2ec4b20ab72971cce4dcf1c; no voters: 814c38565bcd48eeb9de69fe0b86f6ea
I20260928 02:07:46.478474 16689 raft_consensus.cc:2781] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:07:46.478883 16689 raft_consensus.cc:686] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 1 LEADER]: Becoming Leader. State: Replica: 8374a3ebc2ec4b20ab72971cce4dcf1c, State: Running, Role: LEADER
I20260928 02:07:46.479444 16689 consensus_queue.cc:237] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } } peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } }
I20260928 02:07:46.483946 16212 catalog_manager.cc:5495] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8374a3ebc2ec4b20ab72971cce4dcf1c (127.1.182.131). New cstate: current_term: 1 leader_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } health_report { overall_health: HEALTHY } } }
I20260928 02:07:46.518680 16680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.513386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49806 (local address 127.1.182.131:33905)
0928 02:07:46.513567 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:46.513571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:46.513587 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:07:46.513671 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:46.513674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:46.513744 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:46.513864 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:07:46.514211 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.514933 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.516452 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.516912 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.517005 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.517467 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:07:46.517999 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:07:46.518124 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:46.518301 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 02:07:46.518363 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:07:46.526526 16354 raft_consensus.cc:1261] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Refusing update from remote peer 8374a3ebc2ec4b20ab72971cce4dcf1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:07:46.526556 16487 raft_consensus.cc:1261] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c [term 1 FOLLOWER]: Refusing update from remote peer 8374a3ebc2ec4b20ab72971cce4dcf1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:07:46.527364 16694 consensus_queue.cc:1035] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:07:46.527675 16666 tablet.cc:2369] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:07:46.527817 16689 consensus_queue.cc:1035] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:07:46.544137 16698 mvcc.cc:205] Tried to move back new op lower bound from 7334138947683577856 to 7334138947503476736. Current Snapshot: MvccSnapshot[applied={T|T < 7334138947683577856}]
I20260928 02:07:46.556344 16213 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:07:46.556609 16213 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:07:46.559923 16213 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223: Sending DeleteTablet for 3 replicas of tablet 850dccccfea74309b5d291f384383667
I20260928 02:07:46.561117 16467 tablet_service.cc:1514] Processing DeleteTablet for tablet 850dccccfea74309b5d291f384383667 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:07:46 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:50724
I20260928 02:07:46.561239 16334 tablet_service.cc:1514] Processing DeleteTablet for tablet 850dccccfea74309b5d291f384383667 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:07:46 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:38366
I20260928 02:07:46.561704 16600 tablet_service.cc:1514] Processing DeleteTablet for tablet 850dccccfea74309b5d291f384383667 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:07:46 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:49790
I20260928 02:07:46.562773 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 16250
I20260928 02:07:46.573366 16710 tablet_replica.cc:330] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c: stopping tablet replica
I20260928 02:07:46.574174 16710 raft_consensus.cc:2229] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:07:46.575270 16710 raft_consensus.cc:2258] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:07:46.576373 16708 tablet_replica.cc:330] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c: stopping tablet replica
I20260928 02:07:46.578644 16710 ts_tablet_manager.cc:1905] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:07:46.579177 16708 raft_consensus.cc:2229] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:07:46.579751 16708 raft_consensus.cc:2258] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:07:46.584367 16708 ts_tablet_manager.cc:1905] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:07:46.585319 16678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.564295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38382 (local address 127.1.182.129:34723)
0928 02:07:46.564482 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:46.564489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:46.564509 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:07:46.564568 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:46.564576 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:46.564662 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:46.564758 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:46.566908 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.568106 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.570196 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.570864 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.571002 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.576242 (+ 5240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:46.576265 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:46.576282 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:46.576325 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:46.580700 (+ 4375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:46.581574 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:46.581582 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:46.581586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:46.581751 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:46.582176 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:46.582182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:46.582186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:46.582692 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:46.582993 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:46.584664 (+ 1671us) server_negotiation.cc:299] Negotiation successful
0928 02:07:46.584891 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:07:46.586084 16395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:07:46.589051 16710 ts_tablet_manager.cc:1918] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:07:46.589340 16710 log.cc:1196] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/850dccccfea74309b5d291f384383667
I20260928 02:07:46.590155 16710 ts_tablet_manager.cc:1939] T 850dccccfea74309b5d291f384383667 P 8374a3ebc2ec4b20ab72971cce4dcf1c: Deleting consensus metadata
I20260928 02:07:46.593050 16197 catalog_manager.cc:4841] TS 8374a3ebc2ec4b20ab72971cce4dcf1c (127.1.182.131:33905): tablet 850dccccfea74309b5d291f384383667 (table test-table [id=8a61308fa1034f8cafa738832d026c13]) successfully deleted
I20260928 02:07:46.601001 16708 ts_tablet_manager.cc:1918] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:07:46.601279 16708 log.cc:1196] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/850dccccfea74309b5d291f384383667
I20260928 02:07:46.601871 16708 ts_tablet_manager.cc:1939] T 850dccccfea74309b5d291f384383667 P 305b82159a2e4591b5d7597a4620846c: Deleting consensus metadata
I20260928 02:07:46.603767 16197 catalog_manager.cc:4841] TS 305b82159a2e4591b5d7597a4620846c (127.1.182.130:45131): tablet 850dccccfea74309b5d291f384383667 (table test-table [id=8a61308fa1034f8cafa738832d026c13]) successfully deleted
I20260928 02:07:46.689803 16709 tablet_replica.cc:330] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea: stopping tablet replica
I20260928 02:07:46.690037 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 16250
W20260928 02:07:46.708036 16200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.182.129:34723 (error 108)
I20260928 02:07:46.708559 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 16404
I20260928 02:07:46.710479 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.709937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:34723 (local address 127.0.0.1:38392)
0928 02:07:46.710148 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:46.710286 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 02:07:46.710716 16200 catalog_manager.cc:4584] TS 814c38565bcd48eeb9de69fe0b86f6ea (127.1.182.129:34723): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 850dccccfea74309b5d291f384383667: Network error: Client connection negotiation failed: client connection to 127.1.182.129:34723: connect: Connection refused (error 111)
I20260928 02:07:46.728927 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.709471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50736 (local address 127.1.182.130:45131)
0928 02:07:46.709665 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:46.709671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:46.709692 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:07:46.710207 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:46.710212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:46.710296 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:46.710404 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:46.711891 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.712942 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.714213 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.714836 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.714971 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.721734 (+ 6763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:46.721753 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:46.721774 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:46.721834 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:46.725768 (+ 3934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:46.726497 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:46.726508 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:46.726513 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:46.726697 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:46.727127 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:46.727133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:46.727137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:46.727644 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:46.727976 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:46.728224 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 02:07:46.728519 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:07:46.729614 16529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:07:46.764164 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.763709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:34723 (local address 127.0.0.1:38396)
0928 02:07:46.763920 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:46.764009 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 02:07:46.765234 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.764880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:34723 (local address 127.0.0.1:38412)
0928 02:07:46.765037 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:46.765111 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 02:07:46.825723 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.825275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:34723 (local address 127.0.0.1:38426)
0928 02:07:46.825474 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:46.825560 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 02:07:46.826757 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.826429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:34723 (local address 127.0.0.1:38438)
0928 02:07:46.826582 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:46.826651 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 02:07:46.893565 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 16404
I20260928 02:07:46.907873 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.907199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:34723 (local address 127.0.0.1:38446)
0928 02:07:46.907482 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:46.907628 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 02:07:46.909341 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.908781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:34723 (local address 127.0.0.1:38448)
0928 02:07:46.909065 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:46.909176 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260928 02:07:46.913221 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 16537
I20260928 02:07:46.930317 16680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:46.914079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.1.182.131:33905)
0928 02:07:46.914306 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:46.914313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:46.914347 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:07:46.914532 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:46.914538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:46.914636 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:46.914774 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:46.916379 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.917722 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.919363 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:46.920144 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:46.920314 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:46.922800 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:46.922819 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:46.922852 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:46.922913 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:46.927229 (+ 4316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:46.927927 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:46.927937 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:46.927942 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:46.928137 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:46.928498 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:46.928503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:46.928507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:46.929039 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:46.929442 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:46.929648 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 02:07:46.929883 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:07:46.931200 16660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:07:47.054598 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 16537
I20260928 02:07:47.074540 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 16180
I20260928 02:07:47.075771 16243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:07:47.189271 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:47.188728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:34723 (local address 127.0.0.1:38456)
0928 02:07:47.188918 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:47.189048 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 02:07:47.189277 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 16180
I20260928 02:07:47.210877 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:34479
--webserver_interface=127.1.182.190
--webserver_port=45133
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:34479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:07:47.538456 16716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:47.538748 16716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:47.538910 16716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:47.546221 16716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:07:47.546370 16716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:47.546459 16716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:47.546557 16716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:07:47.546646 16716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:07:47.556524 16716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:34479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:34479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=45133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:47.557823 16716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:47.559177 16716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:47.571327 16721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:47.571388 16724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:47.571296 16722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:47.572453 16716 server_base.cc:1034] running on GCE node
I20260928 02:07:47.573302 16716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:47.574735 16716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:47.575960 16716 hybrid_clock.cc:648] HybridClock initialized: now 1790561267575906 us; error 51 us; skew 500 ppm
I20260928 02:07:47.576318 16716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:47 dist-test-slave-23bp krb5kdc[15624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561267, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:47.579437 16716 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:07:47.580871 16716 webserver.cc:460] Webserver started at http://127.1.182.190:45133/ using document root <none> and password file <none>
I20260928 02:07:47.581605 16716 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:47.581751 16716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:07:47.587347 16716 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:07:47.590135 16731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:47.591606 16716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:07:47.591794 16716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "317c6ee7d88947eb8b8ae4731321b223"
format_stamp: "Formatted at 2026-09-28 02:07:41 on dist-test-slave-23bp"
server_key: "b63465a1e8561cce45bee503641a6be1"
server_key_iv: "fffdc50cad54da34f43e612c993ab124"
server_key_version: "encryptionkey@0"
I20260928 02:07:47.592777 16716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:47.612771 16716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:47.616725 16716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:47.617002 16716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:47.635614 16716 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:34479
I20260928 02:07:47.635634 16782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:34479 every 8 connection(s)
I20260928 02:07:47.637117 16716 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:07:47.642692 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 16716
I20260928 02:07:47.643565 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:34723
--local_ip_for_outbound_sockets=127.1.182.129
--tserver_master_addrs=127.1.182.190:34479
--webserver_port=32799
--webserver_interface=127.1.182.129
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:07:47.643944 16783 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:07:47.647753 16783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223: Bootstrap starting.
I20260928 02:07:47.676844 16783 log.cc:824] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223: Log is configured to *not* fsync() on all Append() calls
I20260928 02:07:47.697199 16783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:07:47.697825 16783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223: Bootstrap complete.
I20260928 02:07:47.708757 16783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317c6ee7d88947eb8b8ae4731321b223" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 34479 } }
I20260928 02:07:47.709753 16783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 317c6ee7d88947eb8b8ae4731321b223, State: Initialized, Role: FOLLOWER
I20260928 02:07:47.710451 16783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317c6ee7d88947eb8b8ae4731321b223" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 34479 } }
I20260928 02:07:47.710716 16783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:07:47.710860 16783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:07:47.711051 16783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:07:47.715507 16783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317c6ee7d88947eb8b8ae4731321b223" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 34479 } }
I20260928 02:07:47.716022 16783 leader_election.cc:302] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 317c6ee7d88947eb8b8ae4731321b223; no voters:
I20260928 02:07:47.716645 16783 leader_election.cc:288] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 02:07:47.716845 16787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:07:47.718050 16787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [term 3 LEADER]: Becoming Leader. State: Replica: 317c6ee7d88947eb8b8ae4731321b223, State: Running, Role: LEADER
I20260928 02:07:47.718676 16787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317c6ee7d88947eb8b8ae4731321b223" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 34479 } }
I20260928 02:07:47.718957 16783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:07:47.721506 16789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 317c6ee7d88947eb8b8ae4731321b223. Latest consensus state: current_term: 3 leader_uuid: "317c6ee7d88947eb8b8ae4731321b223" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317c6ee7d88947eb8b8ae4731321b223" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 34479 } } }
I20260928 02:07:47.722014 16789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [sys.catalog]: This master's current role is: LEADER
I20260928 02:07:47.722743 16788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "317c6ee7d88947eb8b8ae4731321b223" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317c6ee7d88947eb8b8ae4731321b223" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 34479 } } }
I20260928 02:07:47.723150 16788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317c6ee7d88947eb8b8ae4731321b223 [sys.catalog]: This master's current role is: LEADER
I20260928 02:07:47.732194 16795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:07:47.743578 16795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:07:47.746130 16795 catalog_manager.cc:1259] Loaded cluster ID: fe7d6cb0497a4cd7aeee849a06f00b76
I20260928 02:07:47.746330 16795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 02:07:47.749214 16795 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0d98e429ad google::LogMessage::Fail() at ??:0
@ 0x7f0d98e4588a google::LogMessage::SendToLog() at ??:0
@ 0x7f0d98e424de google::LogMessage::Flush() at ??:0
@ 0x7f0d98e43a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0dae879512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f0dae876976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0dae8fe610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f0dae8fe439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0d9ba8e345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0d9ba9191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0d9ba91799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0d9ba6ceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0d9ed466db start_thread at ??:0
@ 0x7f0d97cdc61f clone at ??:0
W20260928 02:07:48.042146 16785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:48.042418 16785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:48.042510 16785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:48.049839 16785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:48.049963 16785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:48.050156 16785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:07:48.059661 16785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:34723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=32799
--tserver_master_addrs=127.1.182.190:34479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:48.060966 16785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:48.062325 16785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:48.075071 16824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:48.075059 16825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:48.075127 16827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:48.076222 16785 server_base.cc:1034] running on GCE node
I20260928 02:07:48.076812 16785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:48.077704 16785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:48.078924 16785 hybrid_clock.cc:648] HybridClock initialized: now 1790561268078879 us; error 54 us; skew 500 ppm
I20260928 02:07:48.079288 16785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:48 dist-test-slave-23bp krb5kdc[15624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561268, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:48.082785 16785 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:07:48.084347 16785 webserver.cc:460] Webserver started at http://127.1.182.129:32799/ using document root <none> and password file <none>
I20260928 02:07:48.085089 16785 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:48.085237 16785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:07:48.090608 16785 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:07:48.095899 16834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:48.097239 16785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 02:07:48.097415 16785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "814c38565bcd48eeb9de69fe0b86f6ea"
format_stamp: "Formatted at 2026-09-28 02:07:42 on dist-test-slave-23bp"
server_key: "c259bc7e052e007f91ada4900e261488"
server_key_iv: "44c684cfbf17e6d7767c0646e6210fb3"
server_key_version: "encryptionkey@0"
I20260928 02:07:48.098374 16785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:48.119364 16785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:48.123792 16785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:48.124079 16785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:48.125049 16785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:07:48.126799 16841 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260928 02:07:48.132316 16785 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:07:48.132458 16785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260928 02:07:48.132606 16785 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:07:48.135753 16785 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:07:48.135855 16785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.003s
I20260928 02:07:48.136200 16841 tablet_bootstrap.cc:492] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea: Bootstrap starting.
I20260928 02:07:48.163628 16841 log.cc:824] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea: Log is configured to *not* fsync() on all Append() calls
I20260928 02:07:48.177039 16785 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:34723
I20260928 02:07:48.177039 16948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:34723 every 8 connection(s)
I20260928 02:07:48.177533 16841 tablet_bootstrap.cc:492] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:07:48.178074 16841 tablet_bootstrap.cc:492] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea: Bootstrap complete.
I20260928 02:07:48.178397 16785 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:07:48.179932 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 16785
I20260928 02:07:48.180590 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:45131
--local_ip_for_outbound_sockets=127.1.182.130
--tserver_master_addrs=127.1.182.190:34479
--webserver_port=46751
--webserver_interface=127.1.182.130
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:07:48.179981 16841 ts_tablet_manager.cc:1397] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea: Time spent bootstrapping tablet: real 0.044s user 0.030s sys 0.012s
I20260928 02:07:48.185289 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:48.181391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:48389)
0928 02:07:48.183730 (+ 2339us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:48.184081 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1993,"thread_start_us":1763,"threads_started":1}
W20260928 02:07:48.186065 16949 heartbeater.cc:644] Failed to heartbeat to 127.1.182.190:34479 (0 consecutive failures): Network error: Failed to ping master at 127.1.182.190:34479: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
I20260928 02:07:48.187553 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:48.187097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:37297)
0928 02:07:48.187326 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:48.187418 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260928 02:07:48.189347 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:48.188095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:34625)
0928 02:07:48.189063 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:48.189160 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":873,"thread_start_us":676,"threads_started":1}
W20260928 02:07:48.189540 16949 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:07:48.190445 16841 raft_consensus.cc:348] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } } peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } }
I20260928 02:07:48.191476 16841 raft_consensus.cc:729] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 814c38565bcd48eeb9de69fe0b86f6ea, State: Initialized, Role: FOLLOWER
I20260928 02:07:48.192147 16841 consensus_queue.cc:260] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } } peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } }
I20260928 02:07:48.193737 16841 ts_tablet_manager.cc:1428] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260928 02:07:48.193797 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:48.193377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:35107)
0928 02:07:48.193549 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:48.193648 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 02:07:48.194696 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:48.194392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:58943)
0928 02:07:48.194512 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:48.194578 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 02:07:48.508379 16954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:48.508692 16954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:48.508858 16954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:48.516016 16954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:48.516168 16954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:48.516335 16954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:07:48.525717 16954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:45131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=46751
--tserver_master_addrs=127.1.182.190:34479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:48.527025 16954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:48.528378 16954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:48.541443 16963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:48.541472 16961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:48.541505 16960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:48.542964 16954 server_base.cc:1034] running on GCE node
I20260928 02:07:48.543597 16954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:48.544556 16954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:48.545799 16954 hybrid_clock.cc:648] HybridClock initialized: now 1790561268545762 us; error 72 us; skew 500 ppm
I20260928 02:07:48.546195 16954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:48 dist-test-slave-23bp krb5kdc[15624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561268, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:48.549322 16954 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:07:48.550835 16954 webserver.cc:460] Webserver started at http://127.1.182.130:46751/ using document root <none> and password file <none>
I20260928 02:07:48.551610 16954 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:48.551753 16954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:07:48.556962 16954 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:07:48.559597 16970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:48.560825 16954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:07:48.560998 16954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "305b82159a2e4591b5d7597a4620846c"
format_stamp: "Formatted at 2026-09-28 02:07:42 on dist-test-slave-23bp"
server_key: "8ecac998a08a48ea267e1260040402e1"
server_key_iv: "149eee9f057f5584d28d400f2c307e3d"
server_key_version: "encryptionkey@0"
I20260928 02:07:48.561945 16954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:48.582854 16954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:48.587481 16954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:48.587772 16954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:48.588809 16954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:07:48.590173 16954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:07:48.590294 16954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:48.590438 16954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:07:48.590530 16954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:48.623880 16954 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:45131
I20260928 02:07:48.623966 17083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:45131 every 8 connection(s)
I20260928 02:07:48.625363 16954 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:07:48.625902 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 16954
I20260928 02:07:48.626740 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:33905
--local_ip_for_outbound_sockets=127.1.182.131
--tserver_master_addrs=127.1.182.190:34479
--webserver_port=37413
--webserver_interface=127.1.182.131
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:07:48.631011 17086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:48.628615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.130:41991)
0928 02:07:48.629450 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:48.629775 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":296,"threads_started":1}
W20260928 02:07:48.631757 17084 heartbeater.cc:644] Failed to heartbeat to 127.1.182.190:34479 (0 consecutive failures): Network error: Failed to ping master at 127.1.182.190:34479: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
I20260928 02:07:48.632717 17086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:48.632286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.130:40155)
0928 02:07:48.632474 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:48.632565 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 02:07:48.633671 17086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:48.633244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.130:44945)
0928 02:07:48.633410 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:48.633509 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:07:48.633921 17084 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:07:48.947127 17087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:07:48.947458 17087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:07:48.947646 17087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:07:48.954814 17087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:07:48.954921 17087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:07:48.955052 17087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:07:48.964690 17087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:33905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=37413
--tserver_master_addrs=127.1.182.190:34479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:07:48.966001 17087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:07:48.967338 17087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:07:48.983510 17093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:48.983574 17092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:07:48.983552 17095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:07:48.984378 17087 server_base.cc:1034] running on GCE node
I20260928 02:07:48.984993 17087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:07:48.985992 17087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:07:48.987165 17087 hybrid_clock.cc:648] HybridClock initialized: now 1790561268987116 us; error 41 us; skew 500 ppm
I20260928 02:07:48.987498 17087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:07:48 dist-test-slave-23bp krb5kdc[15624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561268, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:07:48.990638 17087 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:07:48.992137 17087 webserver.cc:460] Webserver started at http://127.1.182.131:37413/ using document root <none> and password file <none>
I20260928 02:07:48.992796 17087 fs_manager.cc:359] Metadata directory not provided
I20260928 02:07:48.992916 17087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:07:48.998670 17087 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260928 02:07:49.001559 17103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:49.002987 17087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:07:49.003208 17087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c"
format_stamp: "Formatted at 2026-09-28 02:07:43 on dist-test-slave-23bp"
server_key: "1cde41b6dbe852885c406b9a222d602c"
server_key_iv: "f15474c4753775cd2242df0521109fe1"
server_key_version: "encryptionkey@0"
I20260928 02:07:49.004171 17087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:07:49.035769 17087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:07:49.039834 17087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:07:49.040117 17087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:07:49.041075 17087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:07:49.042404 17087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:07:49.042514 17087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:49.042647 17087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:07:49.042747 17087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:07:49.075609 17087 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:33905
I20260928 02:07:49.075672 17215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:33905 every 8 connection(s)
I20260928 02:07:49.077021 17087 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:07:49.080076 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 17087
I20260928 02:07:49.081992 17218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:49.079939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.131:51843)
0928 02:07:49.080745 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:49.081036 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":335,"threads_started":1}
W20260928 02:07:49.082617 17216 heartbeater.cc:644] Failed to heartbeat to 127.1.182.190:34479 (0 consecutive failures): Network error: Failed to ping master at 127.1.182.190:34479: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
I20260928 02:07:49.083387 17218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:49.083051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.131:50563)
0928 02:07:49.083226 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:49.083289 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 02:07:49.084121 17218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:49.083828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.131:45895)
0928 02:07:49.083941 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:49.084009 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 02:07:49.084339 17216 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:07:49.196236 17220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:49.195528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:35047)
0928 02:07:49.195943 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:49.196036 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":199,"threads_started":1}
I20260928 02:07:49.635190 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:49.634374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.130:47767)
0928 02:07:49.634793 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:49.634874 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":210,"threads_started":1}
I20260928 02:07:49.958391 17222 raft_consensus.cc:482] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:49.958698 17222 raft_consensus.cc:504] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } } peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } }
I20260928 02:07:49.960343 17222 leader_election.cc:288] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 305b82159a2e4591b5d7597a4620846c (127.1.182.130:45131), 8374a3ebc2ec4b20ab72971cce4dcf1c (127.1.182.131:33905)
Sep 28 02:07:49 dist-test-slave-23bp krb5kdc[15624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561268, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.131@KRBTEST.COM
Sep 28 02:07:49 dist-test-slave-23bp krb5kdc[15624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561268, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.130@KRBTEST.COM
I20260928 02:07:49.983866 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:49.961313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:33905 (local address 127.1.182.129:60461)
0928 02:07:49.963180 (+ 1867us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:49.963197 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:49.963304 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:49.965993 (+ 2689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:49.965997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:49.966895 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:49.967653 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:49.967673 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:49.969212 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:49.969216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:49.970540 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:49.970551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:49.971622 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:49.971630 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:49.971878 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:49.972640 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:49.972663 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:49.975141 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:49.979682 (+ 4541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:49.979690 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:49.979702 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:49.980274 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:49.980737 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:49.980740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:49.980742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:49.980979 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:49.981958 (+ 979us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:49.981966 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:49.982486 (+ 520us) client_negotiation.cc:769] Sending connection context
0928 02:07:49.982799 (+ 313us) client_negotiation.cc:240] Negotiation successful
0928 02:07:49.983388 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1734,"mutex_wait_us":314,"spinlock_wait_cycles":25856,"thread_start_us":1620,"threads_started":1}
I20260928 02:07:49.983901 17226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:49.961609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:60461 (local address 127.1.182.131:33905)
0928 02:07:49.963501 (+ 1892us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:49.963510 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:49.963540 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:07:49.965219 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:49.965231 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:49.965573 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:49.966022 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:49.967855 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:49.969017 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:49.970707 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:49.971362 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:49.971559 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:49.975445 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:49.975457 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:49.975475 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:49.975533 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:49.979536 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:49.980413 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:49.980420 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:49.980426 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:49.980628 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:49.981096 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:49.981099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:49.981101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:49.981808 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:49.982000 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:49.983049 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0928 02:07:49.983484 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1743,"thread_start_us":1627,"threads_started":1}
I20260928 02:07:49.983896 17224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:49.960911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:39011 (local address 127.1.182.130:45131)
0928 02:07:49.962929 (+ 2018us) server_negotiation.cc:206] Beginning negotiation
0928 02:07:49.962948 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:07:49.962997 (+ 49us) server_negotiation.cc:407] Connection header received
0928 02:07:49.964869 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:07:49.964881 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:07:49.965421 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:07:49.965750 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:07:49.967847 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:49.969005 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:49.970671 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:07:49.971362 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:07:49.971588 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:49.975846 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:07:49.975856 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:07:49.975875 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:07:49.975920 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:07:49.979911 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:49.980629 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:49.980637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:49.980643 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:49.980838 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:07:49.981226 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:07:49.981230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:07:49.981233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:07:49.981808 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:07:49.982002 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 02:07:49.983044 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0928 02:07:49.983482 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1817,"thread_start_us":1666,"threads_started":1}
I20260928 02:07:49.983903 17223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:49.960641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:45131 (local address 127.1.182.129:39011)
0928 02:07:49.961260 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:49.961308 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:07:49.962011 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:07:49.965886 (+ 3875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:07:49.965900 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:07:49.966912 (+ 1012us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:07:49.967652 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:49.967673 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:49.969208 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:49.969212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:49.970534 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:07:49.970546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:07:49.971562 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:07:49.971568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:07:49.971884 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:07:49.972692 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:07:49.972709 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:07:49.975526 (+ 2817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:07:49.980037 (+ 4511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:49.980043 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:49.980048 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:49.980508 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:49.980964 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:07:49.980968 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:07:49.980971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:07:49.981125 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:07:49.981958 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:07:49.981966 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:07:49.982486 (+ 520us) client_negotiation.cc:769] Sending connection context
0928 02:07:49.982799 (+ 313us) client_negotiation.cc:240] Negotiation successful
0928 02:07:49.983498 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":385,"spinlock_wait_cycles":50432,"thread_start_us":250,"threads_started":1}
W20260928 02:07:49.987692 16835 leader_election.cc:341] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 305b82159a2e4591b5d7597a4620846c (127.1.182.130:45131): Not found: Tablet not found: 850dccccfea74309b5d291f384383667
W20260928 02:07:49.988025 16835 leader_election.cc:341] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8374a3ebc2ec4b20ab72971cce4dcf1c (127.1.182.131:33905): Not found: Tablet not found: 850dccccfea74309b5d291f384383667
I20260928 02:07:49.988159 16835 leader_election.cc:302] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 814c38565bcd48eeb9de69fe0b86f6ea; no voters: 305b82159a2e4591b5d7597a4620846c, 8374a3ebc2ec4b20ab72971cce4dcf1c
I20260928 02:07:49.988556 17222 raft_consensus.cc:2726] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:07:50.085749 17227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:50.084881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.131:34791)
0928 02:07:50.085349 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:50.085480 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":231,"threads_started":1}
I20260928 02:07:50.197697 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:50.197187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:33863)
0928 02:07:50.197371 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:50.197512 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 02:07:50.636687 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:50.636040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.130:33281)
0928 02:07:50.636453 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:50.636532 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":233,"threads_started":1}
I20260928 02:07:51.087437 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:51.086702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.131:60885)
0928 02:07:51.087191 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:51.087271 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":214,"threads_started":1}
I20260928 02:07:51.199432 17230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:51.198531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:56363)
0928 02:07:51.199062 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:51.199231 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":241,"threads_started":1}
I20260928 02:07:51.638177 17231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:51.637516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.130:42233)
0928 02:07:51.637933 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:51.638010 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":235,"threads_started":1}
I20260928 02:07:52.088904 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:52.088253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.131:48867)
0928 02:07:52.088671 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:52.088747 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":215,"threads_started":1}
I20260928 02:07:52.123238 17233 raft_consensus.cc:482] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:52.123453 17233 raft_consensus.cc:504] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } } peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } }
I20260928 02:07:52.124130 17233 leader_election.cc:288] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 305b82159a2e4591b5d7597a4620846c (127.1.182.130:45131), 8374a3ebc2ec4b20ab72971cce4dcf1c (127.1.182.131:33905)
W20260928 02:07:52.125164 16835 leader_election.cc:341] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8374a3ebc2ec4b20ab72971cce4dcf1c (127.1.182.131:33905): Not found: Tablet not found: 850dccccfea74309b5d291f384383667
W20260928 02:07:52.125388 16835 leader_election.cc:341] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 305b82159a2e4591b5d7597a4620846c (127.1.182.130:45131): Not found: Tablet not found: 850dccccfea74309b5d291f384383667
I20260928 02:07:52.125489 16835 leader_election.cc:302] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 814c38565bcd48eeb9de69fe0b86f6ea; no voters: 305b82159a2e4591b5d7597a4620846c, 8374a3ebc2ec4b20ab72971cce4dcf1c
I20260928 02:07:52.125747 17233 raft_consensus.cc:2726] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:07:52.201124 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:52.200342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:36967)
0928 02:07:52.200810 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:52.200924 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":243,"threads_started":1}
I20260928 02:07:52.640038 17235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:52.639204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.130:40799)
0928 02:07:52.639747 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:52.639828 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":294,"threads_started":1}
I20260928 02:07:53.090554 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:53.089807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.131:52361)
0928 02:07:53.090253 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:53.090334 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":229,"threads_started":1}
I20260928 02:07:53.203486 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:53.202555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:35801)
0928 02:07:53.203134 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:53.203272 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":333,"threads_started":1}
I20260928 02:07:53.641597 17239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:53.640909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.130:51173)
0928 02:07:53.641347 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:53.641425 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":222,"threads_started":1}
I20260928 02:07:54.092199 17240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:54.091477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.131:49575)
0928 02:07:54.091953 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:54.092036 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":266,"threads_started":1}
I20260928 02:07:54.205116 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:54.204414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:46445)
0928 02:07:54.204857 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:54.204936 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":231,"threads_started":1}
I20260928 02:07:54.643278 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:54.642464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.130:39485)
0928 02:07:54.642953 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:54.643033 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":285,"threads_started":1}
I20260928 02:07:54.939205 17243 raft_consensus.cc:482] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:54.939438 17243 raft_consensus.cc:504] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } } peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } }
I20260928 02:07:54.940110 17243 leader_election.cc:288] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 305b82159a2e4591b5d7597a4620846c (127.1.182.130:45131), 8374a3ebc2ec4b20ab72971cce4dcf1c (127.1.182.131:33905)
W20260928 02:07:54.941118 16835 leader_election.cc:341] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8374a3ebc2ec4b20ab72971cce4dcf1c (127.1.182.131:33905): Not found: Tablet not found: 850dccccfea74309b5d291f384383667
W20260928 02:07:54.941315 16835 leader_election.cc:341] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 305b82159a2e4591b5d7597a4620846c (127.1.182.130:45131): Not found: Tablet not found: 850dccccfea74309b5d291f384383667
I20260928 02:07:54.941447 16835 leader_election.cc:302] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 814c38565bcd48eeb9de69fe0b86f6ea; no voters: 305b82159a2e4591b5d7597a4620846c, 8374a3ebc2ec4b20ab72971cce4dcf1c
I20260928 02:07:54.941740 17243 raft_consensus.cc:2726] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:07:55.093809 17244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:55.093107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.131:54659)
0928 02:07:55.093559 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:55.093644 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":237,"threads_started":1}
I20260928 02:07:55.206857 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:55.206006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:58163)
0928 02:07:55.206524 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:55.206627 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":228,"threads_started":1}
I20260928 02:07:55.644896 17246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:55.644118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.130:41811)
0928 02:07:55.644626 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:55.644717 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":220,"threads_started":1}
I20260928 02:07:56.095654 17247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:56.094758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.131:47893)
0928 02:07:56.095393 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:56.095474 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":349,"threads_started":1}
I20260928 02:07:56.208608 17248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:56.207747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:49337)
0928 02:07:56.208283 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:56.208381 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":237,"threads_started":1}
I20260928 02:07:56.503566 17249 raft_consensus.cc:482] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:07:56.503872 17249 raft_consensus.cc:504] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } } peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } }
I20260928 02:07:56.504521 17249 leader_election.cc:288] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 305b82159a2e4591b5d7597a4620846c (127.1.182.130:45131), 8374a3ebc2ec4b20ab72971cce4dcf1c (127.1.182.131:33905)
W20260928 02:07:56.505429 16835 leader_election.cc:341] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 305b82159a2e4591b5d7597a4620846c (127.1.182.130:45131): Not found: Tablet not found: 850dccccfea74309b5d291f384383667
W20260928 02:07:56.505658 16835 leader_election.cc:341] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8374a3ebc2ec4b20ab72971cce4dcf1c (127.1.182.131:33905): Not found: Tablet not found: 850dccccfea74309b5d291f384383667
I20260928 02:07:56.505740 16835 leader_election.cc:302] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 814c38565bcd48eeb9de69fe0b86f6ea; no voters: 305b82159a2e4591b5d7597a4620846c, 8374a3ebc2ec4b20ab72971cce4dcf1c
I20260928 02:07:56.506024 17249 raft_consensus.cc:2726] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:07:56.646554 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:56.645860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.130:41753)
0928 02:07:56.646301 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:56.646382 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":233,"threads_started":1}
I20260928 02:07:57.097210 17251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:57.096549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.131:44105)
0928 02:07:57.096946 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:57.097027 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":208,"threads_started":1}
I20260928 02:07:57.210275 17252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:57.209588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:41901)
0928 02:07:57.210013 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:57.210107 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":219,"threads_started":1}
I20260928 02:07:57.648183 17253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:57.647460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.130:39253)
0928 02:07:57.647928 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:57.648025 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":241,"threads_started":1}
I20260928 02:07:58.098917 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:58.098207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.131:52595)
0928 02:07:58.098684 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:58.098765 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":249,"threads_started":1}
I20260928 02:07:58.212034 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:58.211363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:33513)
0928 02:07:58.211783 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:58.211870 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":220,"threads_started":1}
I20260928 02:07:58.649806 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:58.649112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.130:42539)
0928 02:07:58.649524 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:58.649630 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":222,"threads_started":1}
I20260928 02:07:59.100549 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:59.099860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.131:45757)
0928 02:07:59.100293 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:59.100375 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":248,"threads_started":1}
I20260928 02:07:59.213722 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:59.213016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:50255)
0928 02:07:59.213458 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:59.213543 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":236,"threads_started":1}
I20260928 02:07:59.651494 17259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:07:59.650658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.130:46017)
0928 02:07:59.651218 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 02:07:59.651306 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":247,"threads_started":1}
I20260928 02:08:00.102486 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:00.101573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.131:52707)
0928 02:08:00.102147 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:00.102252 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":245,"threads_started":1}
I20260928 02:08:00.215381 17261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:00.214624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:57801)
0928 02:08:00.215046 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:00.215190 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":211,"threads_started":1}
I20260928 02:08:00.653133 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:00.652400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.130:42289)
0928 02:08:00.652872 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:00.652963 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":236,"threads_started":1}
I20260928 02:08:01.104285 17263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:01.103549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.131:57253)
0928 02:08:01.104014 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:01.104111 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":266,"threads_started":1}
I20260928 02:08:01.217095 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:01.216334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:55317)
0928 02:08:01.216806 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:01.216900 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":242,"threads_started":1}
I20260928 02:08:01.654762 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:01.654033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.130:57775)
0928 02:08:01.654481 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:01.654602 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":228,"threads_started":1}
I20260928 02:08:02.030236 17266 raft_consensus.cc:482] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:08:02.030481 17266 raft_consensus.cc:504] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305b82159a2e4591b5d7597a4620846c" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 45131 } } peers { permanent_uuid: "814c38565bcd48eeb9de69fe0b86f6ea" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34723 } } peers { permanent_uuid: "8374a3ebc2ec4b20ab72971cce4dcf1c" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 33905 } }
I20260928 02:08:02.031265 17266 leader_election.cc:288] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 305b82159a2e4591b5d7597a4620846c (127.1.182.130:45131), 8374a3ebc2ec4b20ab72971cce4dcf1c (127.1.182.131:33905)
W20260928 02:08:02.032191 16835 leader_election.cc:341] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 305b82159a2e4591b5d7597a4620846c (127.1.182.130:45131): Not found: Tablet not found: 850dccccfea74309b5d291f384383667
W20260928 02:08:02.032423 16835 leader_election.cc:341] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8374a3ebc2ec4b20ab72971cce4dcf1c (127.1.182.131:33905): Not found: Tablet not found: 850dccccfea74309b5d291f384383667
I20260928 02:08:02.032546 16835 leader_election.cc:302] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 814c38565bcd48eeb9de69fe0b86f6ea; no voters: 305b82159a2e4591b5d7597a4620846c, 8374a3ebc2ec4b20ab72971cce4dcf1c
I20260928 02:08:02.032806 17266 raft_consensus.cc:2726] T 850dccccfea74309b5d291f384383667 P 814c38565bcd48eeb9de69fe0b86f6ea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:08:02.105882 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:02.105179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.131:38727)
0928 02:08:02.105622 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:02.105714 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":218,"threads_started":1}
I20260928 02:08:02.218784 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:02.218001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:56135)
0928 02:08:02.218469 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:02.218577 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":261,"threads_started":1}
I20260928 02:08:02.656327 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:02.655640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.130:55973)
0928 02:08:02.656051 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:02.656147 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":221,"threads_started":1}
I20260928 02:08:03.107514 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:03.106733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.131:59101)
0928 02:08:03.107228 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:03.107303 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":214,"threads_started":1}
I20260928 02:08:03.220913 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:03.220173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.129:57861)
0928 02:08:03.220644 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:03.220731 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":237,"threads_started":1}
I20260928 02:08:03.658130 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:03.657371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.130:55085)
0928 02:08:03.657840 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:03.657939 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":224,"threads_started":1}
I20260928 02:08:04.082845 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 16785
I20260928 02:08:04.098099 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:04.084214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46244 (local address 127.1.182.129:34723)
0928 02:08:04.084720 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:04.084728 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:04.084755 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:08:04.084823 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:04.084829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:04.085031 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:04.085227 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:08:04.086589 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:04.087721 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:04.088966 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:04.089508 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:04.089609 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:04.092336 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:08:04.092349 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:08:04.092364 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:08:04.092399 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:08:04.095442 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:04.096114 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:04.096121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:04.096123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:04.096286 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:04.096645 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:04.096649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:04.096651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:04.097012 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:08:04.097254 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:04.097484 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 02:08:04.097680 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":232,"threads_started":1}
I20260928 02:08:04.099362 16944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:08:04.109439 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:04.108738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:34479 (local address 127.1.182.131:46429)
0928 02:08:04.109146 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:04.109282 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:34479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":204,"threads_started":1}
I20260928 02:08:04.189528 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 16785
I20260928 02:08:04.207039 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 16954
I20260928 02:08:04.220029 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:04.207867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39040 (local address 127.1.182.130:45131)
0928 02:08:04.208296 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:04.208300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:04.208316 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:08:04.208372 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:04.208376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:04.208478 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:04.208578 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:08:04.209888 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:04.210620 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:04.211867 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:04.212287 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:04.212380 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:04.214865 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:08:04.214869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:08:04.214873 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:08:04.214902 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:08:04.217561 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:04.218254 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:04.218257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:04.218259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:04.218391 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:04.218723 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:04.218727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:04.218729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:04.218972 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:08:04.219306 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:04.219494 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 02:08:04.219653 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":234,"threads_started":1}
I20260928 02:08:04.220799 17079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:08:04.305418 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 16954
I20260928 02:08:04.322669 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 17087
I20260928 02:08:04.337388 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:04.323799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42352 (local address 127.1.182.131:33905)
0928 02:08:04.324400 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:04.324404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:04.324419 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:08:04.324473 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:04.324476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:04.324535 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:04.324624 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:08:04.326415 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:04.327259 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:04.328643 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:04.329053 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:04.329155 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:04.331993 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:08:04.331998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:08:04.332001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:08:04.332033 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:08:04.334763 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:04.335634 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:04.335637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:04.335640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:04.335763 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:04.336136 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:04.336139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:04.336141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:04.336394 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:08:04.336610 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:04.336824 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 02:08:04.336978 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":328,"threads_started":1}
I20260928 02:08:04.338177 17211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:08:04.423939 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 17087
2026-09-28T02:08:04Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24378 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:08:04 dist-test-slave-23bp krb5kdc[17282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:08:04 dist-test-slave-23bp krb5kdc[17282](info): set up 2 sockets
Sep 28 02:08:04 dist-test-slave-23bp krb5kdc[17282](info): commencing operation
krb5kdc: starting...
W20260928 02:08:05.522128 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:08:05 dist-test-slave-23bp krb5kdc[17282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:08:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:08:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:08:05.763698 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:39819
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:39819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:08:06.099541 17295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:08:06.099828 17295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:08:06.099999 17295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:08:06.107256 17295 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:08:06.107349 17295 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:08:06.107430 17295 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:08:06.107530 17295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:08:06.107621 17295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:08:06.107731 17295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:08:06.107805 17295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:08:06.107885 17295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:08:06.117830 17295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:39819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:39819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:08:06.119246 17295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:08:06.120589 17295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:08:06.132560 17300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:06.132464 17303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:06.132524 17301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:08:06.133409 17295 server_base.cc:1034] running on GCE node
I20260928 02:08:06.134099 17295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:08:06.135510 17295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:08:06.136724 17295 hybrid_clock.cc:648] HybridClock initialized: now 1790561286136663 us; error 40 us; skew 500 ppm
I20260928 02:08:06.137065 17295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:08:06 dist-test-slave-23bp krb5kdc[17282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561286, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:08:06.140159 17295 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:08:06.141638 17295 webserver.cc:460] Webserver started at http://127.1.182.190:34935/ using document root <none> and password file <none>
I20260928 02:08:06.142328 17295 fs_manager.cc:359] Metadata directory not provided
I20260928 02:08:06.142460 17295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:08:06.142774 17295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:08:06.145745 17295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "287560f1f14347c6b7f5c3cdfefce8f9"
format_stamp: "Formatted at 2026-09-28 02:08:06 on dist-test-slave-23bp"
server_key: "c1d5a0245fe17104f3b0a385a9885342"
server_key_iv: "3cd82538f37c89406b208db5a39960f8"
server_key_version: "encryptionkey@0"
I20260928 02:08:06.146514 17295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "287560f1f14347c6b7f5c3cdfefce8f9"
format_stamp: "Formatted at 2026-09-28 02:08:06 on dist-test-slave-23bp"
server_key: "c1d5a0245fe17104f3b0a385a9885342"
server_key_iv: "3cd82538f37c89406b208db5a39960f8"
server_key_version: "encryptionkey@0"
I20260928 02:08:06.151592 17295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 02:08:06.155019 17310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:06.156415 17295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:08:06.156694 17295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "287560f1f14347c6b7f5c3cdfefce8f9"
format_stamp: "Formatted at 2026-09-28 02:08:06 on dist-test-slave-23bp"
server_key: "c1d5a0245fe17104f3b0a385a9885342"
server_key_iv: "3cd82538f37c89406b208db5a39960f8"
server_key_version: "encryptionkey@0"
I20260928 02:08:06.156919 17295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:08:06.176369 17295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:08:06.180187 17295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:08:06.180469 17295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:08:06.199111 17295 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:39819
I20260928 02:08:06.199146 17361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:39819 every 8 connection(s)
I20260928 02:08:06.200443 17295 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:08:06.204054 17362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:08:06.208734 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 17295
I20260928 02:08:06.208953 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:08:06.209800 1754 external_mini_cluster.cc:1428] Setting key ebff8a0e75cb5b2ed99a89af83a27968
I20260928 02:08:06.215716 17362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9: Bootstrap starting.
I20260928 02:08:06.219377 17362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9: Neither blocks nor log segments found. Creating new log.
I20260928 02:08:06.220546 17362 log.cc:824] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:08:06.223464 17362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9: No bootstrap required, opened a new log
Sep 28 02:08:06 dist-test-slave-23bp krb5kdc[17282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:08:06.233485 17362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39819 } }
I20260928 02:08:06.233964 17362 raft_consensus.cc:374] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:08:06.234071 17362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287560f1f14347c6b7f5c3cdfefce8f9, State: Initialized, Role: FOLLOWER
I20260928 02:08:06.234264 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:06.211838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57174 (local address 127.1.182.190:39819)
0928 02:08:06.212599 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:06.212610 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:06.212651 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:08:06.214341 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:06.214361 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:06.214732 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:06.215057 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:08:06.216873 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:06.218166 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:06.220182 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:06.221088 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:06.221321 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:06.225986 (+ 4665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:08:06.225995 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:08:06.226013 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:08:06.226048 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:08:06.229394 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:06.230302 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:06.230311 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:06.230319 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:06.230520 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:06.230920 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:06.230925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:06.230929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:06.231779 (+ 850us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:08:06.232104 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:06.232453 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 02:08:06.232894 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":349,"threads_started":1}
I20260928 02:08:06.234754 17362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39819 } }
I20260928 02:08:06.235093 17362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:08:06.235245 17362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:08:06.235419 17362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:08:06.238013 17362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39819 } }
I20260928 02:08:06.238456 17362 leader_election.cc:302] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 287560f1f14347c6b7f5c3cdfefce8f9; no voters:
I20260928 02:08:06.238981 17362 leader_election.cc:288] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:08:06.239142 17367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:08:06.239840 17367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 1 LEADER]: Becoming Leader. State: Replica: 287560f1f14347c6b7f5c3cdfefce8f9, State: Running, Role: LEADER
I20260928 02:08:06.240401 17367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39819 } }
I20260928 02:08:06.241107 17362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:08:06.244987 17368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39819 } } }
I20260928 02:08:06.245438 17368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:08:06.246174 17375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:08:06.246141 17369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 287560f1f14347c6b7f5c3cdfefce8f9. Latest consensus state: current_term: 1 leader_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39819 } } }
I20260928 02:08:06.246517 17369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:08:06.251488 17375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:08:06.260488 17375 catalog_manager.cc:1347] Generated new cluster ID: 92804625726441028b1cbf40711da5b9
I20260928 02:08:06.260610 17375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:08:06.275274 17375 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:08:06.276926 17375 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:08:06.288185 17375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9: Generated new TSK 0
I20260928 02:08:06.288997 17375 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.182.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
I20260928 02:08:06.358606 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39819
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:08:06.687386 17389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:08:06.687654 17389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:08:06.687744 17389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:08:06.694664 17389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:08:06.694789 17389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:08:06.694901 17389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:08:06.704694 17389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:08:06.705937 17389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:08:06.707190 17389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:08:06.720369 17397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:06.720476 17394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:06.720446 17395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:08:06.721446 17389 server_base.cc:1034] running on GCE node
I20260928 02:08:06.722105 17389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:08:06.723150 17389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:08:06.724383 17389 hybrid_clock.cc:648] HybridClock initialized: now 1790561286724343 us; error 49 us; skew 500 ppm
I20260928 02:08:06.724718 17389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:08:06 dist-test-slave-23bp krb5kdc[17282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561286, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:08:06.727818 17389 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:08:06.729269 17389 webserver.cc:460] Webserver started at http://127.1.182.129:41595/ using document root <none> and password file <none>
I20260928 02:08:06.729947 17389 fs_manager.cc:359] Metadata directory not provided
I20260928 02:08:06.730098 17389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:08:06.730417 17389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:08:06.733639 17389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "fba0fbc52616461c9445f8143f74d127"
format_stamp: "Formatted at 2026-09-28 02:08:06 on dist-test-slave-23bp"
server_key: "a7a20a7150c35719f2a9009a3a4c66a0"
server_key_iv: "e2fcb717ae1700e3693d1f4693a2733b"
server_key_version: "encryptionkey@0"
I20260928 02:08:06.734376 17389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "fba0fbc52616461c9445f8143f74d127"
format_stamp: "Formatted at 2026-09-28 02:08:06 on dist-test-slave-23bp"
server_key: "a7a20a7150c35719f2a9009a3a4c66a0"
server_key_iv: "e2fcb717ae1700e3693d1f4693a2733b"
server_key_version: "encryptionkey@0"
I20260928 02:08:06.739508 17389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 02:08:06.742909 17404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:06.744237 17389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:08:06.744413 17389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "fba0fbc52616461c9445f8143f74d127"
format_stamp: "Formatted at 2026-09-28 02:08:06 on dist-test-slave-23bp"
server_key: "a7a20a7150c35719f2a9009a3a4c66a0"
server_key_iv: "e2fcb717ae1700e3693d1f4693a2733b"
server_key_version: "encryptionkey@0"
I20260928 02:08:06.744624 17389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:08:06.765106 17389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:08:06.769045 17389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:08:06.769343 17389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:08:06.770292 17389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:08:06.771643 17389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:08:06.771747 17389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:06.771901 17389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:08:06.771981 17389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:06.804407 17389 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:37921
I20260928 02:08:06.804463 17517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:37921 every 8 connection(s)
I20260928 02:08:06.805716 17389 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:08:06.815850 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 17389
I20260928 02:08:06.816085 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:08:06.816781 1754 external_mini_cluster.cc:1428] Setting key 8d88205b7ae97d33d8832ab010664c8a
Sep 28 02:08:06 dist-test-slave-23bp krb5kdc[17282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561286, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:08:06.827049 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:06.808251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:53065 (local address 127.1.182.190:39819)
0928 02:08:06.808705 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:06.808709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:06.809646 (+ 937us) server_negotiation.cc:407] Connection header received
0928 02:08:06.811605 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:06.811610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:06.811706 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:06.811827 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:08:06.813343 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:06.814100 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:06.815904 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:06.816419 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:06.816561 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:06.820821 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:08:06.820825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:08:06.820828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:08:06.820859 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:08:06.823679 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:06.824711 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:06.824715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:06.824718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:06.824836 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:06.825339 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:06.825342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:06.825344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:06.825610 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:08:06.825718 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:06.826639 (+ 921us) server_negotiation.cc:299] Negotiation successful
0928 02:08:06.826810 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":246,"threads_started":1}
I20260928 02:08:06.828359 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:06.808660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.129:53065)
0928 02:08:06.809461 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:06.809503 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:08:06.811319 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:08:06.812084 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:08:06.812093 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:08:06.812597 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:08:06.813165 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:06.813182 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:06.814249 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:06.814254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:06.815746 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:06.815759 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:06.816653 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:06.816662 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:06.816933 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:06.817674 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:08:06.817714 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:08:06.820612 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:08:06.823845 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:08:06.823857 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:08:06.823878 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:08:06.824556 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:08:06.824955 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:08:06.824963 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:08:06.824969 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:08:06.825224 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:08:06.825743 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:08:06.825753 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:08:06.826233 (+ 480us) client_negotiation.cc:769] Sending connection context
0928 02:08:06.826538 (+ 305us) client_negotiation.cc:240] Negotiation successful
0928 02:08:06.827170 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":353,"threads_started":1}
I20260928 02:08:06.830120 17518 heartbeater.cc:347] Connected to a master server at 127.1.182.190:39819
I20260928 02:08:06.830427 17518 heartbeater.cc:464] Registering TS with master...
I20260928 02:08:06.831346 17518 heartbeater.cc:511] Master 127.1.182.190:39819 requested a full tablet report, sending...
I20260928 02:08:06.833504 17327 ts_manager.cc:194] Registered new tserver with Master: fba0fbc52616461c9445f8143f74d127 (127.1.182.129:37921)
I20260928 02:08:06.834987 17327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:53065
WARNING: no policy specified for kudu/127.1.182.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
I20260928 02:08:06.872251 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39819
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:08:07.194068 17524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:08:07.194392 17524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:08:07.194563 17524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:08:07.201851 17524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:08:07.201997 17524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:08:07.202234 17524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:08:07.212431 17524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:08:07.213672 17524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:08:07.214963 17524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:08:07.228222 17532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:07.228376 17529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:07.228197 17530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:08:07.229095 17524 server_base.cc:1034] running on GCE node
I20260928 02:08:07.229619 17524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:08:07.230436 17524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:08:07.231630 17524 hybrid_clock.cc:648] HybridClock initialized: now 1790561287231598 us; error 32 us; skew 500 ppm
I20260928 02:08:07.232007 17524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:08:07 dist-test-slave-23bp krb5kdc[17282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561287, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:08:07.235020 17524 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:08:07.236424 17524 webserver.cc:460] Webserver started at http://127.1.182.130:44375/ using document root <none> and password file <none>
I20260928 02:08:07.237103 17524 fs_manager.cc:359] Metadata directory not provided
I20260928 02:08:07.237234 17524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:08:07.237541 17524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:08:07.240414 17524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "56c474eeae6b476cb09ef84973019f4d"
format_stamp: "Formatted at 2026-09-28 02:08:07 on dist-test-slave-23bp"
server_key: "964d2e5a817e0cef59f01aeb8641583f"
server_key_iv: "2d7dd619b74af3dc987f4e6cce4732c5"
server_key_version: "encryptionkey@0"
I20260928 02:08:07.241190 17524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "56c474eeae6b476cb09ef84973019f4d"
format_stamp: "Formatted at 2026-09-28 02:08:07 on dist-test-slave-23bp"
server_key: "964d2e5a817e0cef59f01aeb8641583f"
server_key_iv: "2d7dd619b74af3dc987f4e6cce4732c5"
server_key_version: "encryptionkey@0"
I20260928 02:08:07.246523 17524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:08:07.249934 17539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:07.251258 17524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 02:08:07.251452 17524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "56c474eeae6b476cb09ef84973019f4d"
format_stamp: "Formatted at 2026-09-28 02:08:07 on dist-test-slave-23bp"
server_key: "964d2e5a817e0cef59f01aeb8641583f"
server_key_iv: "2d7dd619b74af3dc987f4e6cce4732c5"
server_key_version: "encryptionkey@0"
I20260928 02:08:07.251653 17524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:08:07.265632 17524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:08:07.271371 17524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:08:07.271697 17524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:08:07.272872 17524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:08:07.274312 17524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:08:07.274384 17524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:07.274520 17524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:08:07.274605 17524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:07.307907 17524 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:39855
I20260928 02:08:07.307981 17651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:39855 every 8 connection(s)
I20260928 02:08:07.309181 17524 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:08:07.314236 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 17524
I20260928 02:08:07.314440 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:08:07.314989 1754 external_mini_cluster.cc:1428] Setting key bc670470ab5426c573da30c1ac6b7215
Sep 28 02:08:07 dist-test-slave-23bp krb5kdc[17282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561287, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:08:07.330056 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:07.311677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:46155 (local address 127.1.182.190:39819)
0928 02:08:07.311862 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:07.311866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:07.313144 (+ 1278us) server_negotiation.cc:407] Connection header received
0928 02:08:07.315316 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:07.315321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:07.315411 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:07.315523 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:08:07.317483 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.318592 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.320176 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.320914 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.321052 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:07.324387 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:08:07.324396 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:08:07.324401 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:08:07.324438 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:08:07.327000 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:07.327747 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:07.327752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:07.327756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:07.327888 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:07.328354 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:07.328359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:07.328362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:07.328625 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:08:07.328795 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:07.329476 (+ 681us) server_negotiation.cc:299] Negotiation successful
0928 02:08:07.329720 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:08:07.330992 17654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:07.312099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.130:46155)
0928 02:08:07.312922 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:07.312977 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:08:07.314965 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:08:07.315798 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:08:07.315810 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:08:07.316534 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:08:07.317319 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:07.317338 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.318735 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.318738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:07.320024 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:07.320035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.321123 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.321128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:07.321316 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:07.321998 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:08:07.322020 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:08:07.324164 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:08:07.327188 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:08:07.327197 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:08:07.327215 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:08:07.327623 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:08:07.328031 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:08:07.328036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:08:07.328040 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:08:07.328237 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:08:07.328755 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:08:07.328762 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:08:07.329117 (+ 355us) client_negotiation.cc:769] Sending connection context
0928 02:08:07.329378 (+ 261us) client_negotiation.cc:240] Negotiation successful
0928 02:08:07.329900 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":341,"threads_started":1}
I20260928 02:08:07.332636 17652 heartbeater.cc:347] Connected to a master server at 127.1.182.190:39819
I20260928 02:08:07.332953 17652 heartbeater.cc:464] Registering TS with master...
I20260928 02:08:07.333691 17652 heartbeater.cc:511] Master 127.1.182.190:39819 requested a full tablet report, sending...
I20260928 02:08:07.335155 17327 ts_manager.cc:194] Registered new tserver with Master: 56c474eeae6b476cb09ef84973019f4d (127.1.182.130:39855)
I20260928 02:08:07.335963 17327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:46155
WARNING: no policy specified for kudu/127.1.182.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
I20260928 02:08:07.369764 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39819
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:08:07.698966 17657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:08:07.699272 17657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:08:07.699368 17657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:08:07.706538 17657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:08:07.706640 17657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:08:07.706746 17657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:08:07.716138 17657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:39819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:08:07.717331 17657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:08:07.718538 17657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:08:07.731555 17665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:07.731602 17663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:07.731887 17662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:08:07.732574 17657 server_base.cc:1034] running on GCE node
I20260928 02:08:07.733251 17657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:08:07.734166 17657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:08:07.735385 17657 hybrid_clock.cc:648] HybridClock initialized: now 1790561287735316 us; error 66 us; skew 500 ppm
I20260928 02:08:07.735690 17657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:08:07 dist-test-slave-23bp krb5kdc[17282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561287, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:08:07.738818 17657 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:08:07.740290 17657 webserver.cc:460] Webserver started at http://127.1.182.131:35009/ using document root <none> and password file <none>
I20260928 02:08:07.740967 17657 fs_manager.cc:359] Metadata directory not provided
I20260928 02:08:07.741073 17657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:08:07.741482 17657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:08:07.744483 17657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "d8e99f10ce32441db3ead9ed7901c894"
format_stamp: "Formatted at 2026-09-28 02:08:07 on dist-test-slave-23bp"
server_key: "e64e0f65a0038e1169946e29d6475f49"
server_key_iv: "a57fb90c42ee8918b5766c3a08954278"
server_key_version: "encryptionkey@0"
I20260928 02:08:07.745230 17657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "d8e99f10ce32441db3ead9ed7901c894"
format_stamp: "Formatted at 2026-09-28 02:08:07 on dist-test-slave-23bp"
server_key: "e64e0f65a0038e1169946e29d6475f49"
server_key_iv: "a57fb90c42ee8918b5766c3a08954278"
server_key_version: "encryptionkey@0"
I20260928 02:08:07.750667 17657 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 02:08:07.754101 17672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:07.755505 17657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:08:07.755676 17657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "d8e99f10ce32441db3ead9ed7901c894"
format_stamp: "Formatted at 2026-09-28 02:08:07 on dist-test-slave-23bp"
server_key: "e64e0f65a0038e1169946e29d6475f49"
server_key_iv: "a57fb90c42ee8918b5766c3a08954278"
server_key_version: "encryptionkey@0"
I20260928 02:08:07.755867 17657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:08:07.771409 17657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:08:07.775895 17657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:08:07.776211 17657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:08:07.777186 17657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:08:07.778481 17657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:08:07.778589 17657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:07.778751 17657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:08:07.778827 17657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:07.812261 17657 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:45615
I20260928 02:08:07.812302 17785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:45615 every 8 connection(s)
I20260928 02:08:07.813612 17657 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:08:07.823194 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 17657
I20260928 02:08:07.823390 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:08:07.823909 1754 external_mini_cluster.cc:1428] Setting key cc64254f8a29a43b43be4403fc6d7563
Sep 28 02:08:07 dist-test-slave-23bp krb5kdc[17282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561287, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:08:07.833323 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:07.816059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:41715 (local address 127.1.182.190:39819)
0928 02:08:07.816243 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:07.816248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:07.817365 (+ 1117us) server_negotiation.cc:407] Connection header received
0928 02:08:07.819127 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:07.819133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:07.819216 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:07.819306 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:08:07.820712 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.821465 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.823100 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.823650 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.823745 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:07.827708 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:08:07.827712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:08:07.827716 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:08:07.827749 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:08:07.830345 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:07.831019 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:07.831023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:07.831025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:07.831205 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:07.831659 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:07.831667 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:07.831673 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:07.831954 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:08:07.832060 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:07.832772 (+ 712us) server_negotiation.cc:299] Negotiation successful
0928 02:08:07.832933 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:08:07.834383 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:07.816501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.131:41715)
0928 02:08:07.817201 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:07.817238 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:08:07.818829 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:08:07.819511 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:08:07.819519 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:08:07.819994 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:08:07.820568 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:07.820585 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.821595 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.821599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:07.822920 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:07.822936 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.823866 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.823873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:07.824191 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:07.824964 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:08:07.824986 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:08:07.827524 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:08:07.830474 (+ 2950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:08:07.830481 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:08:07.830494 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:08:07.830917 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:08:07.831319 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:08:07.831324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:08:07.831326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:08:07.831534 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:08:07.832093 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:08:07.832100 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:08:07.832434 (+ 334us) client_negotiation.cc:769] Sending connection context
0928 02:08:07.832678 (+ 244us) client_negotiation.cc:240] Negotiation successful
0928 02:08:07.833258 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":266,"threads_started":1}
I20260928 02:08:07.835966 17786 heartbeater.cc:347] Connected to a master server at 127.1.182.190:39819
I20260928 02:08:07.836277 17786 heartbeater.cc:464] Registering TS with master...
I20260928 02:08:07.836987 17786 heartbeater.cc:511] Master 127.1.182.190:39819 requested a full tablet report, sending...
I20260928 02:08:07.837999 17518 heartbeater.cc:503] Master 127.1.182.190:39819 was elected leader, sending a full tablet report...
I20260928 02:08:07.838512 17327 ts_manager.cc:194] Registered new tserver with Master: d8e99f10ce32441db3ead9ed7901c894 (127.1.182.131:45615)
I20260928 02:08:07.839743 17327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:41715
I20260928 02:08:07.839979 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:08:07.858946 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:07.845360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.1.182.190:39819)
0928 02:08:07.845516 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:07.845521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:07.846055 (+ 534us) server_negotiation.cc:407] Connection header received
0928 02:08:07.846211 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:07.846215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:07.846268 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:07.846348 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:08:07.848112 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.849086 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.850623 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.851129 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.851242 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:07.853819 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:08:07.853824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:08:07.853827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:08:07.853856 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:08:07.856650 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:07.857331 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:07.857335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:07.857337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:07.857458 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:07.857797 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:07.857801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:07.857802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:07.858104 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:08:07.858287 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:07.858575 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 02:08:07.858732 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:08:07.863292 17327 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:08:07.871208 17327 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:08:07.910564 17800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:07.883542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42100 (local address 127.1.182.131:45615)
0928 02:08:07.903783 (+ 20241us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:07.903791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:07.903823 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:08:07.903920 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:07.903929 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:07.904110 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:07.904312 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:08:07.904848 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.906011 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.908330 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.909347 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.909474 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:07.909728 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:07.910035 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 02:08:07.910202 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20061,"thread_start_us":19871,"threads_started":1}
I20260928 02:08:07.910653 17798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:07.883207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:45615 (local address 127.0.0.1:42100)
0928 02:08:07.899715 (+ 16508us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:07.899803 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0928 02:08:07.900013 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:08:07.904292 (+ 4279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:08:07.904297 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:08:07.904325 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:08:07.904687 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:07.904696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.906170 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.906175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:07.908172 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:07.908184 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.909545 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.909551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:07.909766 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:07.909822 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 02:08:07.909967 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 02:08:07.910180 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16230,"mutex_wait_us":27,"thread_start_us":16071,"threads_started":1}
I20260928 02:08:07.912401 17799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:07.883515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:37921 (local address 127.0.0.1:53078)
0928 02:08:07.901991 (+ 18476us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:07.902015 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:08:07.902265 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:08:07.903324 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:08:07.903334 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:08:07.903378 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:08:07.903790 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:07.903806 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.905988 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.905994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:07.909009 (+ 3015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:07.909030 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.911564 (+ 2534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.911570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:07.911784 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:07.911817 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:08:07.911891 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:08:07.912029 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18280,"mutex_wait_us":158,"thread_start_us":18113,"threads_started":1}
I20260928 02:08:07.912588 17803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:07.885225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60640 (local address 127.1.182.130:39855)
0928 02:08:07.901444 (+ 16219us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:07.901456 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:07.902845 (+ 1389us) server_negotiation.cc:407] Connection header received
0928 02:08:07.902974 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:07.902986 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:07.903362 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:07.903573 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:08:07.904513 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.906184 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.909240 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.910843 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.911021 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:07.911288 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:07.912026 (+ 738us) server_negotiation.cc:299] Negotiation successful
0928 02:08:07.912231 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16028,"thread_start_us":15875,"threads_started":1}
I20260928 02:08:07.912926 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:07.884109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.1.182.129:37921)
0928 02:08:07.900874 (+ 16765us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:07.900887 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:07.902146 (+ 1259us) server_negotiation.cc:407] Connection header received
0928 02:08:07.902458 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:07.902469 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:07.902753 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:07.902934 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:08:07.904000 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.905817 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.909206 (+ 3389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.910964 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.911136 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:07.911383 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:07.912433 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0928 02:08:07.912613 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16527,"thread_start_us":16331,"threads_started":1}
I20260928 02:08:07.913074 17802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:07.885042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:39855 (local address 127.0.0.1:60640)
0928 02:08:07.902478 (+ 17436us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:07.902504 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:08:07.902683 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:08:07.903747 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:08:07.903752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:08:07.903784 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:08:07.904250 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:07.904265 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.906370 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.906376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:07.909060 (+ 2684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:07.909074 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:07.911134 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:07.911142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:07.911332 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:07.911400 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 02:08:07.911520 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 02:08:07.911687 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17265,"mutex_wait_us":72,"thread_start_us":17039,"threads_started":1}
I20260928 02:08:07.915591 17720 tablet_service.cc:1467] Processing CreateTablet for tablet 9422bd1f37af475c9c369c72fdcf4234 (DEFAULT_TABLE table=test-table [id=e24346d0c54442afaeb93c5cb87c67f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:08:07.916501 17452 tablet_service.cc:1467] Processing CreateTablet for tablet 9422bd1f37af475c9c369c72fdcf4234 (DEFAULT_TABLE table=test-table [id=e24346d0c54442afaeb93c5cb87c67f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:08:07.917332 17720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9422bd1f37af475c9c369c72fdcf4234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:08:07.917852 17452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9422bd1f37af475c9c369c72fdcf4234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:08:07.918593 17587 tablet_service.cc:1467] Processing CreateTablet for tablet 9422bd1f37af475c9c369c72fdcf4234 (DEFAULT_TABLE table=test-table [id=e24346d0c54442afaeb93c5cb87c67f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:08:07.920049 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9422bd1f37af475c9c369c72fdcf4234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:08:07.929005 17804 tablet_bootstrap.cc:492] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127: Bootstrap starting.
I20260928 02:08:07.932950 17804 tablet_bootstrap.cc:654] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127: Neither blocks nor log segments found. Creating new log.
I20260928 02:08:07.933917 17805 tablet_bootstrap.cc:492] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894: Bootstrap starting.
I20260928 02:08:07.934322 17804 log.cc:824] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127: Log is configured to *not* fsync() on all Append() calls
I20260928 02:08:07.935648 17806 tablet_bootstrap.cc:492] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d: Bootstrap starting.
I20260928 02:08:07.938669 17805 tablet_bootstrap.cc:654] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894: Neither blocks nor log segments found. Creating new log.
I20260928 02:08:07.938750 17806 tablet_bootstrap.cc:654] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d: Neither blocks nor log segments found. Creating new log.
I20260928 02:08:07.940164 17805 log.cc:824] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894: Log is configured to *not* fsync() on all Append() calls
I20260928 02:08:07.940267 17806 log.cc:824] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:08:07.942526 17804 tablet_bootstrap.cc:492] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127: No bootstrap required, opened a new log
I20260928 02:08:07.942801 17804 ts_tablet_manager.cc:1397] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260928 02:08:07.943594 17806 tablet_bootstrap.cc:492] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d: No bootstrap required, opened a new log
I20260928 02:08:07.943608 17805 tablet_bootstrap.cc:492] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894: No bootstrap required, opened a new log
I20260928 02:08:07.943877 17805 ts_tablet_manager.cc:1397] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260928 02:08:07.943873 17806 ts_tablet_manager.cc:1397] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260928 02:08:07.950955 17804 raft_consensus.cc:348] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:07.951411 17804 raft_consensus.cc:374] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:08:07.951550 17804 raft_consensus.cc:729] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba0fbc52616461c9445f8143f74d127, State: Initialized, Role: FOLLOWER
I20260928 02:08:07.952190 17804 consensus_queue.cc:260] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:07.954056 17804 ts_tablet_manager.cc:1428] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260928 02:08:07.956096 17806 raft_consensus.cc:348] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:07.956075 17805 raft_consensus.cc:348] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:07.956593 17805 raft_consensus.cc:374] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:08:07.956619 17806 raft_consensus.cc:374] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:08:07.956722 17805 raft_consensus.cc:729] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8e99f10ce32441db3ead9ed7901c894, State: Initialized, Role: FOLLOWER
I20260928 02:08:07.956722 17806 raft_consensus.cc:729] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c474eeae6b476cb09ef84973019f4d, State: Initialized, Role: FOLLOWER
I20260928 02:08:07.957394 17806 consensus_queue.cc:260] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:07.957394 17805 consensus_queue.cc:260] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:07.959059 17786 heartbeater.cc:503] Master 127.1.182.190:39819 was elected leader, sending a full tablet report...
I20260928 02:08:07.959542 17806 ts_tablet_manager.cc:1428] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
I20260928 02:08:07.959944 17805 ts_tablet_manager.cc:1428] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260928 02:08:07.960466 17652 heartbeater.cc:503] Master 127.1.182.190:39819 was elected leader, sending a full tablet report...
I20260928 02:08:08.003237 17812 raft_consensus.cc:482] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:08:08.003505 17812 raft_consensus.cc:504] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:08.005110 17812 leader_election.cc:288] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fba0fbc52616461c9445f8143f74d127 (127.1.182.129:37921), 56c474eeae6b476cb09ef84973019f4d (127.1.182.130:39855)
I20260928 02:08:08.011683 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:08.005869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:36033 (local address 127.1.182.129:37921)
0928 02:08:08.006187 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:08.006195 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:08.006214 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:08:08.006292 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:08.006298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:08.006371 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:08.006480 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:08:08.006959 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:08.007911 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:08.009787 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:08.010760 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:08.010879 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:08.010930 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:08.011296 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 02:08:08.011410 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260928 02:08:08.011703 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:08.005414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:37921 (local address 127.1.182.131:36033)
0928 02:08:08.005740 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:08.005762 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:08:08.005893 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:08:08.006550 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:08:08.006555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:08:08.006590 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:08:08.006813 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:08.006826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:08.008046 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:08.008051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:08.009624 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:08.009636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:08.010912 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:08.010919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:08.011133 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:08.011173 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:08:08.011277 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:08:08.011423 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260928 02:08:08.012324 17803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:08.005691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:48763 (local address 127.1.182.130:39855)
0928 02:08:08.005884 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:08.005890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:08.006557 (+ 667us) server_negotiation.cc:407] Connection header received
0928 02:08:08.006703 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:08.006709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:08.006798 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:08.006910 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:08:08.007548 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:08.008438 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:08.010298 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:08.011326 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:08.011461 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:08.011524 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:08.011920 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 02:08:08.012059 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:08:08.012862 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:08.005623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:39855 (local address 127.1.182.131:48763)
0928 02:08:08.006439 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:08.006457 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:08:08.006567 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:08:08.006959 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:08:08.006964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:08:08.006990 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:08:08.007393 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:08.007405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:08.008619 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:08.008622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:08.010132 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:08.010147 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:08.011526 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:08.011532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:08.011717 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:08.011756 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 02:08:08.012430 (+ 674us) client_negotiation.cc:240] Negotiation successful
0928 02:08:08.012577 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":1,"thread_start_us":474,"threads_started":1}
I20260928 02:08:08.012924 17472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9422bd1f37af475c9c369c72fdcf4234" candidate_uuid: "d8e99f10ce32441db3ead9ed7901c894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fba0fbc52616461c9445f8143f74d127" is_pre_election: true
I20260928 02:08:08.013615 17472 raft_consensus.cc:2454] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8e99f10ce32441db3ead9ed7901c894 in term 0.
I20260928 02:08:08.013828 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9422bd1f37af475c9c369c72fdcf4234" candidate_uuid: "d8e99f10ce32441db3ead9ed7901c894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c474eeae6b476cb09ef84973019f4d" is_pre_election: true
I20260928 02:08:08.014416 17607 raft_consensus.cc:2454] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8e99f10ce32441db3ead9ed7901c894 in term 0.
I20260928 02:08:08.014379 17673 leader_election.cc:302] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8e99f10ce32441db3ead9ed7901c894, fba0fbc52616461c9445f8143f74d127; no voters:
I20260928 02:08:08.014837 17812 raft_consensus.cc:2781] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:08:08.015012 17812 raft_consensus.cc:482] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:08:08.015170 17812 raft_consensus.cc:3037] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:08:08.017357 17812 raft_consensus.cc:504] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:08.017988 17812 leader_election.cc:288] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 [CANDIDATE]: Term 1 election: Requested vote from peers fba0fbc52616461c9445f8143f74d127 (127.1.182.129:37921), 56c474eeae6b476cb09ef84973019f4d (127.1.182.130:39855)
I20260928 02:08:08.018481 17472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9422bd1f37af475c9c369c72fdcf4234" candidate_uuid: "d8e99f10ce32441db3ead9ed7901c894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fba0fbc52616461c9445f8143f74d127"
I20260928 02:08:08.018481 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9422bd1f37af475c9c369c72fdcf4234" candidate_uuid: "d8e99f10ce32441db3ead9ed7901c894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c474eeae6b476cb09ef84973019f4d"
I20260928 02:08:08.018685 17607 raft_consensus.cc:3037] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:08:08.018685 17472 raft_consensus.cc:3037] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:08:08.021920 17607 raft_consensus.cc:2454] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8e99f10ce32441db3ead9ed7901c894 in term 1.
I20260928 02:08:08.021920 17472 raft_consensus.cc:2454] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8e99f10ce32441db3ead9ed7901c894 in term 1.
I20260928 02:08:08.022339 17676 leader_election.cc:302] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56c474eeae6b476cb09ef84973019f4d, d8e99f10ce32441db3ead9ed7901c894; no voters:
I20260928 02:08:08.022670 17812 raft_consensus.cc:2781] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:08:08.023213 17812 raft_consensus.cc:686] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 [term 1 LEADER]: Becoming Leader. State: Replica: d8e99f10ce32441db3ead9ed7901c894, State: Running, Role: LEADER
I20260928 02:08:08.023778 17812 consensus_queue.cc:237] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:08.028191 17325 catalog_manager.cc:5495] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8e99f10ce32441db3ead9ed7901c894 (127.1.182.131). New cstate: current_term: 1 leader_uuid: "d8e99f10ce32441db3ead9ed7901c894" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } health_report { overall_health: UNKNOWN } } }
W20260928 02:08:08.058079 17519 tablet.cc:2369] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:08:08.061105 17653 tablet.cc:2369] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:08:08.065222 17787 tablet.cc:2369] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:08:08.066393 17800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:08.060803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42108 (local address 127.1.182.131:45615)
0928 02:08:08.061022 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:08.061029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:08.061131 (+ 102us) server_negotiation.cc:407] Connection header received
0928 02:08:08.061384 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:08.061390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:08.061480 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:08.061607 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:08:08.062099 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:08.062787 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:08.064239 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:08.064611 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:08.064701 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:08.065278 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:08:08.065815 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:08:08.065928 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:08.066034 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 02:08:08.066108 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:08:08.074307 17607 raft_consensus.cc:1261] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d [term 1 FOLLOWER]: Refusing update from remote peer d8e99f10ce32441db3ead9ed7901c894: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:08:08.074307 17472 raft_consensus.cc:1261] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Refusing update from remote peer d8e99f10ce32441db3ead9ed7901c894: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:08:08.075393 17814 consensus_queue.cc:1035] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:08:08.075762 17812 consensus_queue.cc:1035] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:08:08.087590 17817 mvcc.cc:205] Tried to move back new op lower bound from 7334139035942191104 to 7334139035749601280. Current Snapshot: MvccSnapshot[applied={T|T < 7334139035942191104}]
I20260928 02:08:08.087848 17819 mvcc.cc:205] Tried to move back new op lower bound from 7334139035942191104 to 7334139035749601280. Current Snapshot: MvccSnapshot[applied={T|T < 7334139035942191104}]
I20260928 02:08:08.090129 17818 mvcc.cc:205] Tried to move back new op lower bound from 7334139035942191104 to 7334139035749601280. Current Snapshot: MvccSnapshot[applied={T|T < 7334139035942191104}]
I20260928 02:08:08.104403 17325 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:08:08.104668 17325 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:08:08.108072 17325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9: Sending DeleteTablet for 3 replicas of tablet 9422bd1f37af475c9c369c72fdcf4234
I20260928 02:08:08.109202 17720 tablet_service.cc:1514] Processing DeleteTablet for tablet 9422bd1f37af475c9c369c72fdcf4234 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:08:08 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:42100
I20260928 02:08:08.109237 17452 tablet_service.cc:1514] Processing DeleteTablet for tablet 9422bd1f37af475c9c369c72fdcf4234 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:08:08 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:53078
I20260928 02:08:08.109352 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet 9422bd1f37af475c9c369c72fdcf4234 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:08:08 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:60640
I20260928 02:08:08.110272 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 17389
I20260928 02:08:08.119658 17828 tablet_replica.cc:330] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127: stopping tablet replica
I20260928 02:08:08.119844 17827 tablet_replica.cc:330] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894: stopping tablet replica
I20260928 02:08:08.120085 17829 tablet_replica.cc:330] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d: stopping tablet replica
I20260928 02:08:08.120200 17828 raft_consensus.cc:2229] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:08:08.120361 17827 raft_consensus.cc:2229] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:08:08.120760 17828 raft_consensus.cc:2258] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:08:08.121230 17827 raft_consensus.cc:2258] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:08:08.123158 17828 ts_tablet_manager.cc:1905] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:08:08.123358 17827 ts_tablet_manager.cc:1905] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:08:08.130280 17828 ts_tablet_manager.cc:1918] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:08:08.130693 17829 raft_consensus.cc:2229] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d [term 1 FOLLOWER]: Raft consensus shutting down.
Sep 28 02:08:08 dist-test-slave-23bp krb5kdc[17282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:08:08.131361 17827 ts_tablet_manager.cc:1918] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:08:08.131425 17829 raft_consensus.cc:2258] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:08:08.131660 17827 log.cc:1196] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/9422bd1f37af475c9c369c72fdcf4234
I20260928 02:08:08.132513 17827 ts_tablet_manager.cc:1939] T 9422bd1f37af475c9c369c72fdcf4234 P d8e99f10ce32441db3ead9ed7901c894: Deleting consensus metadata
I20260928 02:08:08.133589 17829 ts_tablet_manager.cc:1905] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:08:08.134647 17313 catalog_manager.cc:4841] TS d8e99f10ce32441db3ead9ed7901c894 (127.1.182.131:45615): tablet 9422bd1f37af475c9c369c72fdcf4234 (table test-table [id=e24346d0c54442afaeb93c5cb87c67f2]) successfully deleted
I20260928 02:08:08.135279 17828 log.cc:1196] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/wal/wals/9422bd1f37af475c9c369c72fdcf4234
I20260928 02:08:08.135891 17828 ts_tablet_manager.cc:1939] T 9422bd1f37af475c9c369c72fdcf4234 P fba0fbc52616461c9445f8143f74d127: Deleting consensus metadata
I20260928 02:08:08.137717 17311 catalog_manager.cc:4841] TS fba0fbc52616461c9445f8143f74d127 (127.1.182.129:37921): tablet 9422bd1f37af475c9c369c72fdcf4234 (table test-table [id=e24346d0c54442afaeb93c5cb87c67f2]) successfully deleted
I20260928 02:08:08.139782 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:08.111672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53092 (local address 127.1.182.129:37921)
0928 02:08:08.111890 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:08.111899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:08.118661 (+ 6762us) server_negotiation.cc:407] Connection header received
0928 02:08:08.119690 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:08.119698 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:08.119784 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:08.119874 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:08:08.122681 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:08.123787 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:08.125774 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:08.126432 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:08.126526 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:08.132362 (+ 5836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:08:08.132380 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:08:08.132402 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:08:08.132453 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:08:08.136527 (+ 4074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:08.137427 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:08.137435 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:08.137438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:08.137600 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:08.138070 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:08.138089 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:08.138092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:08.138442 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:08:08.138628 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:08.139176 (+ 548us) server_negotiation.cc:299] Negotiation successful
0928 02:08:08.139388 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:08:08.140455 17513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:08:08.140527 17829 ts_tablet_manager.cc:1918] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:08:08.140712 17829 log.cc:1196] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/9422bd1f37af475c9c369c72fdcf4234
I20260928 02:08:08.141306 17829 ts_tablet_manager.cc:1939] T 9422bd1f37af475c9c369c72fdcf4234 P 56c474eeae6b476cb09ef84973019f4d: Deleting consensus metadata
I20260928 02:08:08.143831 17314 catalog_manager.cc:4841] TS 56c474eeae6b476cb09ef84973019f4d (127.1.182.130:39855): tablet 9422bd1f37af475c9c369c72fdcf4234 (table test-table [id=e24346d0c54442afaeb93c5cb87c67f2]) successfully deleted
I20260928 02:08:08.231515 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 17389
I20260928 02:08:08.251250 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 17524
Sep 28 02:08:08 dist-test-slave-23bp krb5kdc[17282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.130@KRBTEST.COM
I20260928 02:08:08.268687 17803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:08.252137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60646 (local address 127.1.182.130:39855)
0928 02:08:08.252316 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:08.252320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:08.252338 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:08:08.252486 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:08.252490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:08.252566 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:08.252649 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:08:08.254398 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:08.255240 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:08.257219 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:08.257615 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:08.257714 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:08.262768 (+ 5054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:08:08.262782 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:08:08.262798 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:08:08.262831 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:08:08.265795 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:08.266469 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:08.266482 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:08.266484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:08.266618 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:08.267011 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:08.267017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:08.267020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:08.267491 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:08:08.267797 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:08.268138 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 02:08:08.268353 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:08:08.269366 17647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:08:08.368695 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 17524
I20260928 02:08:08.386361 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 17657
Sep 28 02:08:08 dist-test-slave-23bp krb5kdc[17282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.131@KRBTEST.COM
I20260928 02:08:08.404798 17800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:08.387224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55324 (local address 127.1.182.131:45615)
0928 02:08:08.387396 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:08.387400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:08.387478 (+ 78us) server_negotiation.cc:407] Connection header received
0928 02:08:08.387673 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:08.387677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:08.387737 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:08.387824 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:08:08.389669 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:08.390457 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:08.392445 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:08.392919 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:08.393045 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:08.398506 (+ 5461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:08:08.398517 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:08:08.398546 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:08:08.398581 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:08:08.401825 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:08.402552 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:08.402563 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:08.402565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:08.402717 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:08.403217 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:08.403221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:08.403223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:08.403593 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:08:08.403930 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:08.404226 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 02:08:08.404437 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:08:08.405442 17781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:08:08.528126 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 17657
I20260928 02:08:08.548354 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 17295
I20260928 02:08:08.549535 17357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:08:08.659601 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 17295
I20260928 02:08:28.681241 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:39819
--webserver_interface=127.1.182.190
--webserver_port=34935
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:39819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:08:29.016820 17835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:08:29.017172 17835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:08:29.017336 17835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:08:29.024539 17835 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:08:29.024673 17835 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:08:29.024766 17835 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:08:29.024861 17835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:08:29.024958 17835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:08:29.025084 17835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:08:29.025169 17835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:08:29.025249 17835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:08:29.034986 17835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:39819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:39819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=34935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:08:29.036388 17835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:08:29.037781 17835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:08:29.060350 17842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:29.060328 17843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:29.060294 17845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:08:29.060719 17835 server_base.cc:1034] running on GCE node
I20260928 02:08:29.061901 17835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:08:29.063561 17835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:08:29.064790 17835 hybrid_clock.cc:648] HybridClock initialized: now 1790561309064721 us; error 37 us; skew 500 ppm
I20260928 02:08:29.065200 17835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:08:29 dist-test-slave-23bp krb5kdc[17282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561309, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:08:29.074294 17835 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:08:29.076038 17835 webserver.cc:460] Webserver started at http://127.1.182.190:34935/ using document root <none> and password file <none>
I20260928 02:08:29.076849 17835 fs_manager.cc:359] Metadata directory not provided
I20260928 02:08:29.076985 17835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:08:29.082995 17835 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:08:29.086071 17852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:29.087648 17835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 02:08:29.087867 17835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "287560f1f14347c6b7f5c3cdfefce8f9"
format_stamp: "Formatted at 2026-09-28 02:08:06 on dist-test-slave-23bp"
server_key: "c1d5a0245fe17104f3b0a385a9885342"
server_key_iv: "3cd82538f37c89406b208db5a39960f8"
server_key_version: "encryptionkey@0"
I20260928 02:08:29.089210 17835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:08:29.136024 17835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:08:29.141399 17835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:08:29.141767 17835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:08:29.163631 17835 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:39819
I20260928 02:08:29.163679 17903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:39819 every 8 connection(s)
I20260928 02:08:29.165339 17835 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:08:29.165992 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 17835
I20260928 02:08:29.167042 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:37921
--local_ip_for_outbound_sockets=127.1.182.129
--tserver_master_addrs=127.1.182.190:39819
--webserver_port=41595
--webserver_interface=127.1.182.129
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:08:29.173524 17904 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:08:29.176931 17904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9: Bootstrap starting.
I20260928 02:08:29.196920 17904 log.cc:824] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:08:29.209261 17904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:08:29.209856 17904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9: Bootstrap complete.
I20260928 02:08:29.219357 17904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39819 } }
I20260928 02:08:29.220257 17904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287560f1f14347c6b7f5c3cdfefce8f9, State: Initialized, Role: FOLLOWER
I20260928 02:08:29.220952 17904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39819 } }
I20260928 02:08:29.221210 17904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:08:29.221343 17904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:08:29.221529 17904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:08:29.225353 17904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39819 } }
I20260928 02:08:29.225859 17904 leader_election.cc:302] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 287560f1f14347c6b7f5c3cdfefce8f9; no voters:
I20260928 02:08:29.226545 17904 leader_election.cc:288] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:08:29.226689 17908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:08:29.227874 17908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 2 LEADER]: Becoming Leader. State: Replica: 287560f1f14347c6b7f5c3cdfefce8f9, State: Running, Role: LEADER
I20260928 02:08:29.228541 17908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39819 } }
I20260928 02:08:29.228912 17904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:08:29.231257 17910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 287560f1f14347c6b7f5c3cdfefce8f9. Latest consensus state: current_term: 2 leader_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39819 } } }
I20260928 02:08:29.231297 17909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39819 } } }
I20260928 02:08:29.231674 17910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:08:29.231732 17909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:08:29.240623 17915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:08:29.250176 17915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:08:29.252050 17915 catalog_manager.cc:1259] Loaded cluster ID: 92804625726441028b1cbf40711da5b9
I20260928 02:08:29.252218 17915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:08:29.257153 17915 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:08:29.259203 17915 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9: Loaded TSK: 0
I20260928 02:08:29.289261 17925 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:08:29.307370 17915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9: Generated new TSK 1
I20260928 02:08:29.307811 17915 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:08:29.533833 17906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:08:29.534097 17906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:08:29.534247 17906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:08:29.541405 17906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:08:29.541510 17906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:08:29.541671 17906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:08:29.551575 17906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:37921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=41595
--tserver_master_addrs=127.1.182.190:39819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:08:29.552860 17906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:08:29.554214 17906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:08:29.567927 17932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:29.568039 17931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:29.568572 17934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:08:29.569231 17906 server_base.cc:1034] running on GCE node
I20260928 02:08:29.569886 17906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:08:29.570847 17906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:08:29.572075 17906 hybrid_clock.cc:648] HybridClock initialized: now 1790561309572026 us; error 41 us; skew 500 ppm
I20260928 02:08:29.572397 17906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:08:29 dist-test-slave-23bp krb5kdc[17282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561309, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:08:29.575554 17906 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:08:29.577126 17906 webserver.cc:460] Webserver started at http://127.1.182.129:41595/ using document root <none> and password file <none>
I20260928 02:08:29.577821 17906 fs_manager.cc:359] Metadata directory not provided
I20260928 02:08:29.577958 17906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:08:29.583567 17906 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:08:29.586607 17941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:29.588414 17906 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:08:29.588613 17906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "fba0fbc52616461c9445f8143f74d127"
format_stamp: "Formatted at 2026-09-28 02:08:06 on dist-test-slave-23bp"
server_key: "a7a20a7150c35719f2a9009a3a4c66a0"
server_key_iv: "e2fcb717ae1700e3693d1f4693a2733b"
server_key_version: "encryptionkey@0"
I20260928 02:08:29.590015 17906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:08:29.640987 17906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:08:29.646586 17906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:08:29.646899 17906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:08:29.647997 17906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:08:29.649243 17906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:08:29.649336 17906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:29.649410 17906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:08:29.649467 17906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:29.682093 17906 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:37921
I20260928 02:08:29.682207 18055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:37921 every 8 connection(s)
I20260928 02:08:29.683631 17906 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:08:29.685009 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 17906
I20260928 02:08:29.685698 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:39855
--local_ip_for_outbound_sockets=127.1.182.130
--tserver_master_addrs=127.1.182.190:39819
--webserver_port=44375
--webserver_interface=127.1.182.130
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:08:29 dist-test-slave-23bp krb5kdc[17282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561309, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:08:29.707942 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:29.686729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:57681 (local address 127.1.182.190:39819)
0928 02:08:29.687439 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:29.687458 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:29.688011 (+ 553us) server_negotiation.cc:407] Connection header received
0928 02:08:29.690129 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:29.690141 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:29.690458 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:29.690836 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:08:29.692565 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:29.693937 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:29.695769 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:29.696504 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:29.696731 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:29.700232 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:08:29.700239 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:08:29.700254 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:08:29.700294 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:08:29.703338 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:29.704121 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:29.704128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:29.704134 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:29.704320 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:29.704758 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:29.704767 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:29.704769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:29.705232 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:08:29.705447 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:29.706221 (+ 774us) server_negotiation.cc:299] Negotiation successful
0928 02:08:29.706614 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":345,"threads_started":1}
I20260928 02:08:29.707942 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:29.687025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.129:57681)
0928 02:08:29.687832 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:29.687875 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:08:29.689590 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:08:29.690963 (+ 1373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:08:29.690977 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:08:29.691618 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:08:29.692319 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:29.692339 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:29.694111 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:29.694119 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:29.695618 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:29.695631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:29.696755 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:29.696763 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:29.697061 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:29.697761 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:08:29.697783 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:08:29.699955 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:08:29.703479 (+ 3524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:08:29.703488 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:08:29.703504 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:08:29.703936 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:08:29.704440 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:08:29.704444 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:08:29.704448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:08:29.704635 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:08:29.705386 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:08:29.705394 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:08:29.705733 (+ 339us) client_negotiation.cc:769] Sending connection context
0928 02:08:29.706000 (+ 267us) client_negotiation.cc:240] Negotiation successful
0928 02:08:29.706562 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":318,"threads_started":1}
I20260928 02:08:29.712005 18056 heartbeater.cc:347] Connected to a master server at 127.1.182.190:39819
I20260928 02:08:29.712332 18056 heartbeater.cc:464] Registering TS with master...
I20260928 02:08:29.713084 18056 heartbeater.cc:511] Master 127.1.182.190:39819 requested a full tablet report, sending...
I20260928 02:08:29.715278 17869 ts_manager.cc:194] Registered new tserver with Master: fba0fbc52616461c9445f8143f74d127 (127.1.182.129:37921)
I20260928 02:08:29.716704 17869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:57681
W20260928 02:08:30.013269 18060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:08:30.013636 18060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:08:30.013814 18060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:08:30.020969 18060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:08:30.021129 18060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:08:30.021317 18060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:08:30.031081 18060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:39855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=44375
--tserver_master_addrs=127.1.182.190:39819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:08:30.032405 18060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:08:30.033720 18060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:08:30.046691 18065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:30.046751 18066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:30.046731 18068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:08:30.047574 18060 server_base.cc:1034] running on GCE node
I20260928 02:08:30.048112 18060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:08:30.049002 18060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:08:30.050201 18060 hybrid_clock.cc:648] HybridClock initialized: now 1790561310050162 us; error 38 us; skew 500 ppm
I20260928 02:08:30.050518 18060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:08:30 dist-test-slave-23bp krb5kdc[17282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561310, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:08:30.053602 18060 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:08:30.055137 18060 webserver.cc:460] Webserver started at http://127.1.182.130:44375/ using document root <none> and password file <none>
I20260928 02:08:30.055810 18060 fs_manager.cc:359] Metadata directory not provided
I20260928 02:08:30.055946 18060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:08:30.061113 18060 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:08:30.063742 18075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:30.065181 18060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:08:30.065366 18060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "56c474eeae6b476cb09ef84973019f4d"
format_stamp: "Formatted at 2026-09-28 02:08:07 on dist-test-slave-23bp"
server_key: "964d2e5a817e0cef59f01aeb8641583f"
server_key_iv: "2d7dd619b74af3dc987f4e6cce4732c5"
server_key_version: "encryptionkey@0"
I20260928 02:08:30.066349 18060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:08:30.081037 18060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:08:30.084957 18060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:08:30.085253 18060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:08:30.086185 18060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:08:30.087559 18060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:08:30.087687 18060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:30.087817 18060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:08:30.087908 18060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:30.121647 18060 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:39855
I20260928 02:08:30.121732 18188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:39855 every 8 connection(s)
I20260928 02:08:30.123194 18060 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:08:30.130630 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 18060
I20260928 02:08:30.131453 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:45615
--local_ip_for_outbound_sockets=127.1.182.131
--tserver_master_addrs=127.1.182.190:39819
--webserver_port=35009
--webserver_interface=127.1.182.131
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:08:30 dist-test-slave-23bp krb5kdc[17282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561310, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:08:30.144865 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:30.125784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:59789 (local address 127.1.182.190:39819)
0928 02:08:30.125988 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:30.125993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:30.127273 (+ 1280us) server_negotiation.cc:407] Connection header received
0928 02:08:30.129135 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:30.129140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:30.129222 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:30.129312 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:08:30.131111 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.132225 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.133821 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.134562 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.134708 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:30.138423 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:08:30.138433 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:08:30.138440 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:08:30.138497 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:08:30.141162 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:30.142140 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:30.142162 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:30.142168 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:30.142393 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:30.142954 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:30.142957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:30.142959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:30.143368 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:08:30.143561 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:30.144432 (+ 871us) server_negotiation.cc:299] Negotiation successful
0928 02:08:30.144619 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:08:30.146536 18191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:30.126297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.130:59789)
0928 02:08:30.127044 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:30.127148 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0928 02:08:30.128872 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:08:30.129592 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:08:30.129601 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:08:30.130221 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:08:30.130897 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:30.130917 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.132401 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.132405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:30.133671 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:30.133683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.134770 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.134775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:30.134985 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:30.135755 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:08:30.135777 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:08:30.138168 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:08:30.141304 (+ 3136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:08:30.141316 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:08:30.141336 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:08:30.142018 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:08:30.142553 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:08:30.142560 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:08:30.142567 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:08:30.142836 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:08:30.143524 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:08:30.143558 (+ 34us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:08:30.144029 (+ 471us) client_negotiation.cc:769] Sending connection context
0928 02:08:30.144357 (+ 328us) client_negotiation.cc:240] Negotiation successful
0928 02:08:30.144960 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":290,"threads_started":1}
I20260928 02:08:30.148205 18189 heartbeater.cc:347] Connected to a master server at 127.1.182.190:39819
I20260928 02:08:30.148527 18189 heartbeater.cc:464] Registering TS with master...
I20260928 02:08:30.149256 18189 heartbeater.cc:511] Master 127.1.182.190:39819 requested a full tablet report, sending...
I20260928 02:08:30.150897 17869 ts_manager.cc:194] Registered new tserver with Master: 56c474eeae6b476cb09ef84973019f4d (127.1.182.130:39855)
I20260928 02:08:30.152236 17869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:59789
W20260928 02:08:30.466799 18192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:08:30.467137 18192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:08:30.467263 18192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:08:30.474670 18192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:08:30.474778 18192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:08:30.474887 18192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:08:30.484687 18192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:45615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=35009
--tserver_master_addrs=127.1.182.190:39819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:08:30.485889 18192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:08:30.487221 18192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:08:30.500425 18198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:30.500655 18197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:30.500401 18200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:08:30.501241 18192 server_base.cc:1034] running on GCE node
I20260928 02:08:30.501785 18192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:08:30.502637 18192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:08:30.503858 18192 hybrid_clock.cc:648] HybridClock initialized: now 1790561310503817 us; error 37 us; skew 500 ppm
I20260928 02:08:30.504173 18192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:08:30 dist-test-slave-23bp krb5kdc[17282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561310, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:08:30.507298 18192 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:08:30.508793 18192 webserver.cc:460] Webserver started at http://127.1.182.131:35009/ using document root <none> and password file <none>
I20260928 02:08:30.509481 18192 fs_manager.cc:359] Metadata directory not provided
I20260928 02:08:30.509644 18192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:08:30.514989 18192 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:08:30.517800 18207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:30.519137 18192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:08:30.519307 18192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "d8e99f10ce32441db3ead9ed7901c894"
format_stamp: "Formatted at 2026-09-28 02:08:07 on dist-test-slave-23bp"
server_key: "e64e0f65a0038e1169946e29d6475f49"
server_key_iv: "a57fb90c42ee8918b5766c3a08954278"
server_key_version: "encryptionkey@0"
I20260928 02:08:30.520267 18192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:08:30.565521 18192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:08:30.569870 18192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:08:30.570159 18192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:08:30.571267 18192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:08:30.572611 18192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:08:30.572727 18192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:30.572854 18192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:08:30.572940 18192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:30.606557 18192 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:45615
I20260928 02:08:30.606616 18320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:45615 every 8 connection(s)
I20260928 02:08:30.608059 18192 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:08:30.608601 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 18192
Sep 28 02:08:30 dist-test-slave-23bp krb5kdc[17282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561310, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:08:30.625200 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:30.609716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54014 (local address 127.1.182.190:39819)
0928 02:08:30.609930 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:30.609939 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:30.610434 (+ 495us) server_negotiation.cc:407] Connection header received
0928 02:08:30.610608 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:30.610612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:30.610677 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:30.610763 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:08:30.612717 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.613538 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.614905 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.615655 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.615787 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:30.619029 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:08:30.619035 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:08:30.619040 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:08:30.619128 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:08:30.622131 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:30.622815 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:30.622822 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:30.622825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:30.622988 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:30.623448 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:30.623455 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:30.623458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:30.623786 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:08:30.624021 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:30.624494 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 02:08:30.624730 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:08:30.631413 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:30.611782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:56231 (local address 127.1.182.190:39819)
0928 02:08:30.612225 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:30.612231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:30.613395 (+ 1164us) server_negotiation.cc:407] Connection header received
0928 02:08:30.615280 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:30.615286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:30.615367 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:30.615474 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:08:30.617025 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.618198 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.619859 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.620671 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.620818 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:30.624682 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:08:30.624688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:08:30.624693 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:08:30.624724 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:08:30.628675 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:30.629318 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:30.629321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:30.629323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:30.629424 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:30.629804 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:30.629807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:30.629809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:30.630146 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:08:30.630254 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:30.630896 (+ 642us) server_negotiation.cc:299] Negotiation successful
0928 02:08:30.631130 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":226,"threads_started":1}
I20260928 02:08:30.632347 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:30.612418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.131:56231)
0928 02:08:30.613238 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:30.613283 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:08:30.614973 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:08:30.615680 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:08:30.615688 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:08:30.616288 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:08:30.616850 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:30.616866 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.618342 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.618346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:30.619704 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:30.619721 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.620916 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.620925 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:30.621216 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:30.621815 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:08:30.621837 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:08:30.624382 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:08:30.628783 (+ 4401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:08:30.628789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:08:30.628802 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:08:30.629208 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:08:30.629524 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:08:30.629527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:08:30.629529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:08:30.629723 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:08:30.630255 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:08:30.630260 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:08:30.630587 (+ 327us) client_negotiation.cc:769] Sending connection context
0928 02:08:30.630801 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 02:08:30.631246 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":343,"threads_started":1}
I20260928 02:08:30.633646 18321 heartbeater.cc:347] Connected to a master server at 127.1.182.190:39819
I20260928 02:08:30.633952 18321 heartbeater.cc:464] Registering TS with master...
I20260928 02:08:30.634670 18321 heartbeater.cc:511] Master 127.1.182.190:39819 requested a full tablet report, sending...
I20260928 02:08:30.636075 17869 ts_manager.cc:194] Registered new tserver with Master: d8e99f10ce32441db3ead9ed7901c894 (127.1.182.131:45615)
I20260928 02:08:30.636876 17869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:56231
I20260928 02:08:30.640129 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:08:30.646823 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:30.640940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54026 (local address 127.1.182.190:39819)
0928 02:08:30.641087 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:30.641092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:30.641644 (+ 552us) server_negotiation.cc:407] Connection header received
0928 02:08:30.641847 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:30.641851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:30.641988 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:30.642109 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:08:30.642540 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.643409 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.644893 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.645295 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.645428 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:30.645941 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:08:30.646166 (+ 225us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 02:08:30.646528 (+ 362us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54026: token expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 02:08:30.646960 18324 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54026: token expired
W20260928 02:08:30.647423 17789 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.182.190:39819)
I20260928 02:08:30.659507 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:30.647875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54036 (local address 127.1.182.190:39819)
0928 02:08:30.648022 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:30.648026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:30.648041 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:08:30.648166 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:30.648169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:30.648220 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:30.648294 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:08:30.649563 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.650239 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.651504 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.651875 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.651964 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:30.654574 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:08:30.654585 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:08:30.654588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:08:30.654617 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:08:30.657193 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:30.657864 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:30.657868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:30.657870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:30.657979 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:30.658297 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:30.658300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:30.658302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:30.658571 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:08:30.658767 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:30.658999 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 02:08:30.659204 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:08:30.662631 17869 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:08:30.670014 17869 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:08:30.709198 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:30.682031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:39855 (local address 127.0.0.1:39118)
0928 02:08:30.700174 (+ 18143us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:30.700192 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:08:30.700296 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:08:30.701518 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:08:30.701523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:08:30.701554 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:08:30.701987 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:30.702000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.703933 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.703938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:30.706147 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:30.706163 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.708095 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.708101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:30.708293 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:30.708356 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 02:08:30.708495 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 02:08:30.708702 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18018,"mutex_wait_us":47,"thread_start_us":16264,"threads_started":1}
I20260928 02:08:30.709340 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:30.681449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42366 (local address 127.1.182.129:37921)
0928 02:08:30.699920 (+ 18471us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:30.699931 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:30.699964 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:08:30.700073 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:30.700081 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:30.700360 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:30.700643 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:08:30.702498 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.704254 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.706897 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.707989 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.708134 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:30.708357 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:30.708773 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 02:08:30.708989 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18273,"thread_start_us":18132,"threads_started":1}
I20260928 02:08:30.709717 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:30.682586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39118 (local address 127.1.182.130:39855)
0928 02:08:30.700952 (+ 18366us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:30.700959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:30.700982 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:08:30.701057 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:30.701062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:30.701253 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:30.701526 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:08:30.702192 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.703800 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.706350 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.707881 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.708029 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:30.708244 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:30.709212 (+ 968us) server_negotiation.cc:299] Negotiation successful
0928 02:08:30.709399 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18113,"thread_start_us":17944,"threads_started":1}
I20260928 02:08:30.710013 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:30.681424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:37921 (local address 127.0.0.1:42366)
0928 02:08:30.697960 (+ 16536us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:30.697989 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 02:08:30.698169 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:08:30.700938 (+ 2769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:08:30.700950 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:08:30.700996 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:08:30.701598 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:30.701620 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.704398 (+ 2778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.704404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:30.706749 (+ 2345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:30.706762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.708409 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.708415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:30.708594 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:30.708623 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 02:08:30.708706 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 02:08:30.708813 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16334,"mutex_wait_us":56,"thread_start_us":10309,"threads_started":1}
I20260928 02:08:30.710362 18333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:30.681401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:45615 (local address 127.0.0.1:58774)
0928 02:08:30.692286 (+ 10885us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:30.692338 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:08:30.692571 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:08:30.703282 (+ 10711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:08:30.703287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:08:30.703312 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:08:30.703620 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:30.703630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.705414 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.705421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:30.707666 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:30.707681 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.709611 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.709618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:30.709760 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:30.709814 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 02:08:30.709909 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 02:08:30.710045 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10540,"thread_start_us":18538,"threads_started":1}
I20260928 02:08:30.711180 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:30.682146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.1.182.131:45615)
0928 02:08:30.702258 (+ 20112us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:30.702268 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:30.702302 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:08:30.702397 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:30.702404 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:30.702628 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:30.703281 (+ 653us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:08:30.703807 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.705253 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.707835 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.709114 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.709233 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:30.709452 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:30.710671 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0928 02:08:30.710835 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19937,"thread_start_us":19722,"threads_started":1}
I20260928 02:08:30.713709 17989 tablet_service.cc:1467] Processing CreateTablet for tablet b3d43cfe06d9423c8b1b153b47a708c5 (DEFAULT_TABLE table=test-table [id=4e15a56ed8574d54a4b2a8b1e44e8832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:08:30.714112 18256 tablet_service.cc:1467] Processing CreateTablet for tablet b3d43cfe06d9423c8b1b153b47a708c5 (DEFAULT_TABLE table=test-table [id=4e15a56ed8574d54a4b2a8b1e44e8832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:08:30.715132 17989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3d43cfe06d9423c8b1b153b47a708c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:08:30.715497 18256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3d43cfe06d9423c8b1b153b47a708c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:08:30.716869 18123 tablet_service.cc:1467] Processing CreateTablet for tablet b3d43cfe06d9423c8b1b153b47a708c5 (DEFAULT_TABLE table=test-table [id=4e15a56ed8574d54a4b2a8b1e44e8832]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:08:30.718109 18123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3d43cfe06d9423c8b1b153b47a708c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:08:30.719522 18056 heartbeater.cc:503] Master 127.1.182.190:39819 was elected leader, sending a full tablet report...
I20260928 02:08:30.728320 18338 tablet_bootstrap.cc:492] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894: Bootstrap starting.
I20260928 02:08:30.731555 18338 tablet_bootstrap.cc:654] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894: Neither blocks nor log segments found. Creating new log.
I20260928 02:08:30.732733 18338 log.cc:824] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894: Log is configured to *not* fsync() on all Append() calls
I20260928 02:08:30.732939 18339 tablet_bootstrap.cc:492] T b3d43cfe06d9423c8b1b153b47a708c5 P 56c474eeae6b476cb09ef84973019f4d: Bootstrap starting.
I20260928 02:08:30.734673 18340 tablet_bootstrap.cc:492] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127: Bootstrap starting.
I20260928 02:08:30.736938 18339 tablet_bootstrap.cc:654] T b3d43cfe06d9423c8b1b153b47a708c5 P 56c474eeae6b476cb09ef84973019f4d: Neither blocks nor log segments found. Creating new log.
I20260928 02:08:30.738147 18340 tablet_bootstrap.cc:654] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127: Neither blocks nor log segments found. Creating new log.
I20260928 02:08:30.738238 18338 tablet_bootstrap.cc:492] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894: No bootstrap required, opened a new log
I20260928 02:08:30.738233 18339 log.cc:824] T b3d43cfe06d9423c8b1b153b47a708c5 P 56c474eeae6b476cb09ef84973019f4d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:08:30.738555 18338 ts_tablet_manager.cc:1397] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20260928 02:08:30.739375 18340 log.cc:824] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127: Log is configured to *not* fsync() on all Append() calls
I20260928 02:08:30.741258 18339 tablet_bootstrap.cc:492] T b3d43cfe06d9423c8b1b153b47a708c5 P 56c474eeae6b476cb09ef84973019f4d: No bootstrap required, opened a new log
I20260928 02:08:30.741514 18339 ts_tablet_manager.cc:1397] T b3d43cfe06d9423c8b1b153b47a708c5 P 56c474eeae6b476cb09ef84973019f4d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 02:08:30.741964 18340 tablet_bootstrap.cc:492] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127: No bootstrap required, opened a new log
I20260928 02:08:30.742210 18340 ts_tablet_manager.cc:1397] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 02:08:30.749378 18338 raft_consensus.cc:348] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:30.749536 18340 raft_consensus.cc:348] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:30.749832 18338 raft_consensus.cc:374] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:08:30.749917 18340 raft_consensus.cc:374] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:08:30.749957 18338 raft_consensus.cc:729] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8e99f10ce32441db3ead9ed7901c894, State: Initialized, Role: FOLLOWER
I20260928 02:08:30.750020 18340 raft_consensus.cc:729] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba0fbc52616461c9445f8143f74d127, State: Initialized, Role: FOLLOWER
I20260928 02:08:30.750638 18340 consensus_queue.cc:260] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:30.750640 18338 consensus_queue.cc:260] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:30.752213 18340 ts_tablet_manager.cc:1428] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260928 02:08:30.752233 18339 raft_consensus.cc:348] T b3d43cfe06d9423c8b1b153b47a708c5 P 56c474eeae6b476cb09ef84973019f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:30.752722 18339 raft_consensus.cc:374] T b3d43cfe06d9423c8b1b153b47a708c5 P 56c474eeae6b476cb09ef84973019f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:08:30.752825 18339 raft_consensus.cc:729] T b3d43cfe06d9423c8b1b153b47a708c5 P 56c474eeae6b476cb09ef84973019f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c474eeae6b476cb09ef84973019f4d, State: Initialized, Role: FOLLOWER
I20260928 02:08:30.753500 18321 heartbeater.cc:503] Master 127.1.182.190:39819 was elected leader, sending a full tablet report...
I20260928 02:08:30.754168 18338 ts_tablet_manager.cc:1428] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
I20260928 02:08:30.754567 18339 consensus_queue.cc:260] T b3d43cfe06d9423c8b1b153b47a708c5 P 56c474eeae6b476cb09ef84973019f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:30.755699 18189 heartbeater.cc:503] Master 127.1.182.190:39819 was elected leader, sending a full tablet report...
I20260928 02:08:30.757457 18339 ts_tablet_manager.cc:1428] T b3d43cfe06d9423c8b1b153b47a708c5 P 56c474eeae6b476cb09ef84973019f4d: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260928 02:08:30.759859 18345 raft_consensus.cc:482] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:08:30.760087 18345 raft_consensus.cc:504] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:30.761615 18345 leader_election.cc:288] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fba0fbc52616461c9445f8143f74d127 (127.1.182.129:37921), 56c474eeae6b476cb09ef84973019f4d (127.1.182.130:39855)
I20260928 02:08:30.767855 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:30.762727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:57051 (local address 127.1.182.129:37921)
0928 02:08:30.762932 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:30.762938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:30.762963 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:08:30.763035 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:30.763039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:30.763148 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:30.763253 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:08:30.763749 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.764434 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.766126 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.766926 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.767118 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:30.767193 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:30.767482 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 02:08:30.767612 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:08:30.767877 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:30.761871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:37921 (local address 127.1.182.131:57051)
0928 02:08:30.762078 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:30.762106 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 02:08:30.762245 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:08:30.763282 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:08:30.763286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:08:30.763308 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:08:30.763585 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:30.763599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.764577 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.764583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:30.765973 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:30.765984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.767121 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.767127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:30.767321 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:30.767348 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 02:08:30.767456 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 02:08:30.767587 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":40}
I20260928 02:08:30.767951 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:30.762085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:39973 (local address 127.1.182.130:39855)
0928 02:08:30.762316 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:30.762325 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:30.762748 (+ 423us) server_negotiation.cc:407] Connection header received
0928 02:08:30.762899 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:30.762902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:30.762954 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:30.763026 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:08:30.763676 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.764383 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.766275 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.767030 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.767248 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:30.767318 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:30.767575 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 02:08:30.767695 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:08:30.768072 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:30.761967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:39855 (local address 127.1.182.131:39973)
0928 02:08:30.762659 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:30.762673 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:08:30.762786 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:08:30.763063 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:08:30.763089 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:08:30.763121 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:08:30.763520 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:30.763534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.764551 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.764556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:30.766128 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:30.766140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.767274 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.767280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:30.767426 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:30.767448 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:08:30.767550 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 02:08:30.767681 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":432,"threads_started":1}
I20260928 02:08:30.769040 18143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3d43cfe06d9423c8b1b153b47a708c5" candidate_uuid: "d8e99f10ce32441db3ead9ed7901c894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c474eeae6b476cb09ef84973019f4d" is_pre_election: true
I20260928 02:08:30.769237 18010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3d43cfe06d9423c8b1b153b47a708c5" candidate_uuid: "d8e99f10ce32441db3ead9ed7901c894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fba0fbc52616461c9445f8143f74d127" is_pre_election: true
I20260928 02:08:30.769575 18143 raft_consensus.cc:2454] T b3d43cfe06d9423c8b1b153b47a708c5 P 56c474eeae6b476cb09ef84973019f4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8e99f10ce32441db3ead9ed7901c894 in term 0.
I20260928 02:08:30.769779 18010 raft_consensus.cc:2454] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8e99f10ce32441db3ead9ed7901c894 in term 0.
I20260928 02:08:30.770354 18208 leader_election.cc:302] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 56c474eeae6b476cb09ef84973019f4d, d8e99f10ce32441db3ead9ed7901c894, fba0fbc52616461c9445f8143f74d127; no voters:
I20260928 02:08:30.770783 18345 raft_consensus.cc:2781] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:08:30.770969 18345 raft_consensus.cc:482] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:08:30.771095 18345 raft_consensus.cc:3037] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:08:30.773311 18345 raft_consensus.cc:504] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:30.774022 18345 leader_election.cc:288] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 [CANDIDATE]: Term 1 election: Requested vote from peers fba0fbc52616461c9445f8143f74d127 (127.1.182.129:37921), 56c474eeae6b476cb09ef84973019f4d (127.1.182.130:39855)
I20260928 02:08:30.774477 18010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3d43cfe06d9423c8b1b153b47a708c5" candidate_uuid: "d8e99f10ce32441db3ead9ed7901c894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fba0fbc52616461c9445f8143f74d127"
I20260928 02:08:30.774497 18143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3d43cfe06d9423c8b1b153b47a708c5" candidate_uuid: "d8e99f10ce32441db3ead9ed7901c894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c474eeae6b476cb09ef84973019f4d"
I20260928 02:08:30.774677 18010 raft_consensus.cc:3037] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:08:30.774686 18143 raft_consensus.cc:3037] T b3d43cfe06d9423c8b1b153b47a708c5 P 56c474eeae6b476cb09ef84973019f4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:08:30.777272 18010 raft_consensus.cc:2454] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8e99f10ce32441db3ead9ed7901c894 in term 1.
I20260928 02:08:30.777298 18143 raft_consensus.cc:2454] T b3d43cfe06d9423c8b1b153b47a708c5 P 56c474eeae6b476cb09ef84973019f4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8e99f10ce32441db3ead9ed7901c894 in term 1.
I20260928 02:08:30.777678 18211 leader_election.cc:302] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56c474eeae6b476cb09ef84973019f4d, d8e99f10ce32441db3ead9ed7901c894; no voters:
I20260928 02:08:30.777946 18345 raft_consensus.cc:2781] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:08:30.778420 18345 raft_consensus.cc:686] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 [term 1 LEADER]: Becoming Leader. State: Replica: d8e99f10ce32441db3ead9ed7901c894, State: Running, Role: LEADER
I20260928 02:08:30.778952 18345 consensus_queue.cc:237] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:30.783511 17867 catalog_manager.cc:5495] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8e99f10ce32441db3ead9ed7901c894 (127.1.182.131). New cstate: current_term: 1 leader_uuid: "d8e99f10ce32441db3ead9ed7901c894" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } health_report { overall_health: UNKNOWN } } }
I20260928 02:08:30.808102 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:30.802825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58788 (local address 127.1.182.131:45615)
0928 02:08:30.803010 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:30.803015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:30.803146 (+ 131us) server_negotiation.cc:407] Connection header received
0928 02:08:30.803346 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:30.803349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:30.803438 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:30.803560 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:08:30.803887 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.804636 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.806083 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.806472 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.806561 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:30.807013 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:08:30.807603 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:08:30.807711 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:30.807813 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 02:08:30.807887 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:08:30.816346 18143 raft_consensus.cc:1261] T b3d43cfe06d9423c8b1b153b47a708c5 P 56c474eeae6b476cb09ef84973019f4d [term 1 FOLLOWER]: Refusing update from remote peer d8e99f10ce32441db3ead9ed7901c894: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:08:30.816346 18010 raft_consensus.cc:1261] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Refusing update from remote peer d8e99f10ce32441db3ead9ed7901c894: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:08:30.817333 18348 consensus_queue.cc:1035] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:08:30.817713 18345 consensus_queue.cc:1035] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:08:30.829156 18350 mvcc.cc:205] Tried to move back new op lower bound from 7334139129093611520 to 7334139128955203584. Current Snapshot: MvccSnapshot[applied={T|T < 7334139129093611520}]
I20260928 02:08:30.844581 17869 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:08:30.844906 17869 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:08:30.848670 17869 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9: Sending DeleteTablet for 3 replicas of tablet b3d43cfe06d9423c8b1b153b47a708c5
I20260928 02:08:30.849877 17989 tablet_service.cc:1514] Processing DeleteTablet for tablet b3d43cfe06d9423c8b1b153b47a708c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:08:30 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:42366
I20260928 02:08:30.849892 18256 tablet_service.cc:1514] Processing DeleteTablet for tablet b3d43cfe06d9423c8b1b153b47a708c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:08:30 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:58774
I20260928 02:08:30.850418 18123 tablet_service.cc:1514] Processing DeleteTablet for tablet b3d43cfe06d9423c8b1b153b47a708c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:08:30 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:39118
I20260928 02:08:30.850858 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 17906
I20260928 02:08:30.852679 18361 tablet_replica.cc:330] T b3d43cfe06d9423c8b1b153b47a708c5 P 56c474eeae6b476cb09ef84973019f4d: stopping tablet replica
I20260928 02:08:30.853252 18361 raft_consensus.cc:2229] T b3d43cfe06d9423c8b1b153b47a708c5 P 56c474eeae6b476cb09ef84973019f4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:08:30.854007 18361 raft_consensus.cc:2258] T b3d43cfe06d9423c8b1b153b47a708c5 P 56c474eeae6b476cb09ef84973019f4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:08:30.860002 18361 ts_tablet_manager.cc:1905] T b3d43cfe06d9423c8b1b153b47a708c5 P 56c474eeae6b476cb09ef84973019f4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:08:30.870937 18359 tablet_replica.cc:330] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127: stopping tablet replica
I20260928 02:08:30.871263 18360 tablet_replica.cc:330] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894: stopping tablet replica
I20260928 02:08:30.871618 18359 raft_consensus.cc:2229] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:08:30.871694 18360 raft_consensus.cc:2229] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:08:30.872457 18360 raft_consensus.cc:2258] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:08:30.875017 18360 ts_tablet_manager.cc:1905] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:08:30.875198 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:30.852320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42380 (local address 127.1.182.129:37921)
0928 02:08:30.852526 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:30.852536 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:30.852560 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:08:30.852621 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:30.852626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:30.852724 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:30.852849 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:08:30.854974 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.856353 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.858868 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.860940 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.861092 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:30.865688 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:08:30.865709 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:08:30.865729 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:08:30.865775 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:08:30.870396 (+ 4621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:30.871498 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:30.871509 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:30.871513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:30.871686 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:30.872188 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:30.872193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:30.872197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:30.872734 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:08:30.873037 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:30.874424 (+ 1387us) server_negotiation.cc:299] Negotiation successful
0928 02:08:30.874693 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:08:30.875543 18359 raft_consensus.cc:2258] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:08:30.876189 18051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:08:30.876641 18361 ts_tablet_manager.cc:1918] T b3d43cfe06d9423c8b1b153b47a708c5 P 56c474eeae6b476cb09ef84973019f4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:08:30.876912 18361 log.cc:1196] T b3d43cfe06d9423c8b1b153b47a708c5 P 56c474eeae6b476cb09ef84973019f4d: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/b3d43cfe06d9423c8b1b153b47a708c5
I20260928 02:08:30.877601 18361 ts_tablet_manager.cc:1939] T b3d43cfe06d9423c8b1b153b47a708c5 P 56c474eeae6b476cb09ef84973019f4d: Deleting consensus metadata
I20260928 02:08:30.880005 17856 catalog_manager.cc:4841] TS 56c474eeae6b476cb09ef84973019f4d (127.1.182.130:39855): tablet b3d43cfe06d9423c8b1b153b47a708c5 (table test-table [id=4e15a56ed8574d54a4b2a8b1e44e8832]) successfully deleted
I20260928 02:08:30.886634 18360 ts_tablet_manager.cc:1918] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:08:30.886852 18360 log.cc:1196] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/b3d43cfe06d9423c8b1b153b47a708c5
I20260928 02:08:30.887504 18360 ts_tablet_manager.cc:1939] T b3d43cfe06d9423c8b1b153b47a708c5 P d8e99f10ce32441db3ead9ed7901c894: Deleting consensus metadata
I20260928 02:08:30.889561 17855 catalog_manager.cc:4841] TS d8e99f10ce32441db3ead9ed7901c894 (127.1.182.131:45615): tablet b3d43cfe06d9423c8b1b153b47a708c5 (table test-table [id=4e15a56ed8574d54a4b2a8b1e44e8832]) successfully deleted
I20260928 02:08:30.973388 18359 ts_tablet_manager.cc:1905] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:08:30.974494 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 17906
W20260928 02:08:30.992606 17853 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.182.129:37921 (error 108)
I20260928 02:08:30.992861 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 18060
I20260928 02:08:30.995102 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:30.994487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:37921 (local address 127.0.0.1:42396)
0928 02:08:30.994747 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:30.994866 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260928 02:08:30.995440 17853 catalog_manager.cc:4584] TS fba0fbc52616461c9445f8143f74d127 (127.1.182.129:37921): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b3d43cfe06d9423c8b1b153b47a708c5: Network error: Client connection negotiation failed: client connection to 127.1.182.129:37921: connect: Connection refused (error 111)
I20260928 02:08:31.008432 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:30.994438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39120 (local address 127.1.182.130:39855)
0928 02:08:30.994657 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:30.994691 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:30.994712 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:08:30.994779 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:30.994784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:30.994855 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:30.995003 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:08:30.996625 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.997375 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.998808 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:30.999427 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:30.999567 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:31.002497 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:08:31.002511 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:08:31.002532 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:08:31.002564 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:08:31.005473 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:31.006214 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:31.006222 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:31.006224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:31.006347 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:31.006733 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:31.006740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:31.006743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:31.007178 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:08:31.007470 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:31.007783 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 02:08:31.008008 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:08:31.009121 18184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:08:31.049237 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:31.048659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:37921 (local address 127.0.0.1:42400)
0928 02:08:31.048915 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:31.049040 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 02:08:31.050460 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:31.050018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:37921 (local address 127.0.0.1:42408)
0928 02:08:31.050192 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:31.050301 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 02:08:31.104419 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 18060
I20260928 02:08:31.111518 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:31.110727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:37921 (local address 127.0.0.1:42416)
0928 02:08:31.111163 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:31.111284 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214}
I20260928 02:08:31.112879 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:31.112412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:37921 (local address 127.0.0.1:42430)
0928 02:08:31.112614 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:31.112714 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 02:08:31.122622 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 18192
I20260928 02:08:31.139813 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:31.123566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58790 (local address 127.1.182.131:45615)
0928 02:08:31.123739 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:31.123749 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:31.123791 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:08:31.124095 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:31.124104 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:31.124224 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:31.124350 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:08:31.125831 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:31.126871 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:31.129048 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:31.129567 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:31.129667 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:31.133371 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:08:31.133382 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:08:31.133397 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:08:31.133437 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:08:31.136556 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:31.137465 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:31.137472 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:31.137474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:31.137633 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:31.138109 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:31.138113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:31.138115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:31.138475 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:08:31.138722 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:31.139337 (+ 615us) server_negotiation.cc:299] Negotiation successful
0928 02:08:31.139504 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:08:31.140583 18316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:08:31.193427 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:31.192901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:37921 (local address 127.0.0.1:42432)
0928 02:08:31.193134 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:31.193245 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 02:08:31.194591 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:31.194125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:37921 (local address 127.0.0.1:42442)
0928 02:08:31.194355 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:31.194460 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260928 02:08:31.268018 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 18192
I20260928 02:08:31.287922 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 17835
I20260928 02:08:31.289079 17899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:08:31.407869 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:31.407280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:37921 (local address 127.0.0.1:42456)
0928 02:08:31.407542 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:31.407674 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:37921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 02:08:31.407894 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 17835
I20260928 02:08:51.430903 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:39819
--webserver_interface=127.1.182.190
--webserver_port=34935
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:39819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:08:51.764927 18367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:08:51.765287 18367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:08:51.765444 18367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:08:51.772763 18367 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:08:51.772897 18367 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:08:51.773000 18367 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:08:51.773093 18367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:08:51.773195 18367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:08:51.773315 18367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:08:51.773402 18367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:08:51.773485 18367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:08:51.783393 18367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:39819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:39819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=34935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:08:51.784724 18367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:08:51.786139 18367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:08:51.798354 18372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:51.798331 18373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:51.798516 18375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:08:51.798768 18367 server_base.cc:1034] running on GCE node
I20260928 02:08:51.799938 18367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:08:51.801312 18367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:08:51.802541 18367 hybrid_clock.cc:648] HybridClock initialized: now 1790561331802472 us; error 44 us; skew 500 ppm
I20260928 02:08:51.802891 18367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:08:51 dist-test-slave-23bp krb5kdc[17282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561331, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:08:51.814457 18367 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:08:51.816211 18367 webserver.cc:460] Webserver started at http://127.1.182.190:34935/ using document root <none> and password file <none>
I20260928 02:08:51.817008 18367 fs_manager.cc:359] Metadata directory not provided
I20260928 02:08:51.817157 18367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:08:51.823211 18367 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260928 02:08:51.826262 18382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:51.827757 18367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 02:08:51.827942 18367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "287560f1f14347c6b7f5c3cdfefce8f9"
format_stamp: "Formatted at 2026-09-28 02:08:06 on dist-test-slave-23bp"
server_key: "c1d5a0245fe17104f3b0a385a9885342"
server_key_iv: "3cd82538f37c89406b208db5a39960f8"
server_key_version: "encryptionkey@0"
I20260928 02:08:51.828943 18367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:08:51.852578 18367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:08:51.857116 18367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:08:51.857414 18367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:08:51.877398 18367 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:39819
I20260928 02:08:51.877421 18433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:39819 every 8 connection(s)
I20260928 02:08:51.878994 18367 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:08:51.885841 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 18367
I20260928 02:08:51.886893 18434 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:08:51.886965 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:37921
--local_ip_for_outbound_sockets=127.1.182.129
--tserver_master_addrs=127.1.182.190:39819
--webserver_port=41595
--webserver_interface=127.1.182.129
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:08:51.892359 18434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9: Bootstrap starting.
I20260928 02:08:51.914523 18434 log.cc:824] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:08:51.930786 18434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:08:51.931399 18434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9: Bootstrap complete.
I20260928 02:08:51.941218 18434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39819 } }
I20260928 02:08:51.942282 18434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287560f1f14347c6b7f5c3cdfefce8f9, State: Initialized, Role: FOLLOWER
I20260928 02:08:51.943044 18434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39819 } }
I20260928 02:08:51.943354 18434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:08:51.943480 18434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:08:51.943666 18434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:08:51.947657 18434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39819 } }
I20260928 02:08:51.948154 18434 leader_election.cc:302] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 287560f1f14347c6b7f5c3cdfefce8f9; no voters:
I20260928 02:08:51.948868 18434 leader_election.cc:288] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 02:08:51.949020 18438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:08:51.950305 18438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [term 3 LEADER]: Becoming Leader. State: Replica: 287560f1f14347c6b7f5c3cdfefce8f9, State: Running, Role: LEADER
I20260928 02:08:51.950958 18438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39819 } }
I20260928 02:08:51.951241 18434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:08:51.953450 18440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 287560f1f14347c6b7f5c3cdfefce8f9. Latest consensus state: current_term: 3 leader_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39819 } } }
I20260928 02:08:51.953814 18440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:08:51.954838 18439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287560f1f14347c6b7f5c3cdfefce8f9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39819 } } }
I20260928 02:08:51.955708 18439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:08:51.958493 18445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:08:51.967490 18445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:08:51.969125 18445 catalog_manager.cc:1259] Loaded cluster ID: 92804625726441028b1cbf40711da5b9
I20260928 02:08:51.969293 18445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:08:51.973459 18445 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:08:51.975243 18445 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9: Loaded TSK: 0
I20260928 02:08:51.975519 18445 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 287560f1f14347c6b7f5c3cdfefce8f9: Loaded TSK: 1
F20260928 02:08:51.977002 18445 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb3a2f909ad google::LogMessage::Fail() at ??:0
@ 0x7fb3a2f9388a google::LogMessage::SendToLog() at ??:0
@ 0x7fb3a2f904de google::LogMessage::Flush() at ??:0
@ 0x7fb3a2f91a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb3a6cf47d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb3a6ce373e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb3b89c31c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb3b89c2783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb3b8a4e2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fb3b8a4e0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fb3b8a985e9 std::function<>::operator()() at ??:0
@ 0x7fb3b89c6cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb3b89c4c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb3b8a4c610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb3b8a4c439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb3a5bdc345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb3a5bdf91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb3a5bdf799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb3a5bbaeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb3a8e946db start_thread at ??:0
@ 0x7fb3a1e2a61f clone at ??:0
W20260928 02:08:52.268949 18436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:08:52.269273 18436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:08:52.269464 18436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:08:52.276865 18436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:08:52.277019 18436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:08:52.277199 18436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:08:52.286895 18436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:37921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=41595
--tserver_master_addrs=127.1.182.190:39819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:08:52.288223 18436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:08:52.289572 18436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:08:52.303102 18485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:52.303174 18483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:52.303279 18482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:08:52.304075 18436 server_base.cc:1034] running on GCE node
I20260928 02:08:52.304675 18436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:08:52.305691 18436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:08:52.306912 18436 hybrid_clock.cc:648] HybridClock initialized: now 1790561332306861 us; error 48 us; skew 500 ppm
I20260928 02:08:52.307273 18436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:08:52 dist-test-slave-23bp krb5kdc[17282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561332, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:08:52.310334 18436 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:08:52.311811 18436 webserver.cc:460] Webserver started at http://127.1.182.129:41595/ using document root <none> and password file <none>
I20260928 02:08:52.312486 18436 fs_manager.cc:359] Metadata directory not provided
I20260928 02:08:52.312651 18436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:08:52.317850 18436 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:08:52.320848 18492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:52.322209 18436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:08:52.322376 18436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "fba0fbc52616461c9445f8143f74d127"
format_stamp: "Formatted at 2026-09-28 02:08:06 on dist-test-slave-23bp"
server_key: "a7a20a7150c35719f2a9009a3a4c66a0"
server_key_iv: "e2fcb717ae1700e3693d1f4693a2733b"
server_key_version: "encryptionkey@0"
I20260928 02:08:52.323412 18436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:08:52.335403 18436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:08:52.339221 18436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:08:52.339478 18436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:08:52.340390 18436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:08:52.342229 18499 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260928 02:08:52.348552 18436 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:08:52.348695 18436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260928 02:08:52.348839 18436 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:08:52.352098 18436 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:08:52.352201 18436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260928 02:08:52.352522 18499 tablet_bootstrap.cc:492] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127: Bootstrap starting.
I20260928 02:08:52.360940 18499 log.cc:824] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127: Log is configured to *not* fsync() on all Append() calls
I20260928 02:08:52.376907 18499 tablet_bootstrap.cc:492] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:08:52.377758 18499 tablet_bootstrap.cc:492] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127: Bootstrap complete.
I20260928 02:08:52.378496 18499 ts_tablet_manager.cc:1397] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127: Time spent bootstrapping tablet: real 0.026s user 0.017s sys 0.008s
I20260928 02:08:52.387935 18499 raft_consensus.cc:348] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:52.389629 18499 raft_consensus.cc:729] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba0fbc52616461c9445f8143f74d127, State: Initialized, Role: FOLLOWER
I20260928 02:08:52.390513 18499 consensus_queue.cc:260] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:52.392138 18499 ts_tablet_manager.cc:1428] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260928 02:08:52.397873 18436 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:37921
I20260928 02:08:52.397964 18607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:37921 every 8 connection(s)
I20260928 02:08:52.399159 18436 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:08:52.403676 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:52.401804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.129:50801)
0928 02:08:52.402529 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:52.402766 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":304,"threads_started":1}
W20260928 02:08:52.404420 18608 heartbeater.cc:644] Failed to heartbeat to 127.1.182.190:39819 (0 consecutive failures): Network error: Failed to ping master at 127.1.182.190:39819: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
I20260928 02:08:52.405223 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:52.404916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.129:44615)
0928 02:08:52.405066 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:52.405129 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 02:08:52.405580 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 18436
I20260928 02:08:52.406081 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:52.405778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.129:43711)
0928 02:08:52.405900 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:52.405966 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 02:08:52.406369 18608 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:08:52.406364 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:39855
--local_ip_for_outbound_sockets=127.1.182.130
--tserver_master_addrs=127.1.182.190:39819
--webserver_port=44375
--webserver_interface=127.1.182.130
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:08:52.739568 18611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:08:52.739838 18611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:08:52.739929 18611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:08:52.747123 18611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:08:52.747231 18611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:08:52.747340 18611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:08:52.756848 18611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:39855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=44375
--tserver_master_addrs=127.1.182.190:39819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:08:52.758097 18611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:08:52.759390 18611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:08:52.772854 18616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:52.772888 18617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:52.772815 18619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:08:52.773869 18611 server_base.cc:1034] running on GCE node
I20260928 02:08:52.774501 18611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:08:52.775535 18611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:08:52.776736 18611 hybrid_clock.cc:648] HybridClock initialized: now 1790561332776691 us; error 41 us; skew 500 ppm
I20260928 02:08:52.777093 18611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:08:52 dist-test-slave-23bp krb5kdc[17282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561332, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:08:52.780208 18611 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:08:52.781713 18611 webserver.cc:460] Webserver started at http://127.1.182.130:44375/ using document root <none> and password file <none>
I20260928 02:08:52.782411 18611 fs_manager.cc:359] Metadata directory not provided
I20260928 02:08:52.782586 18611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:08:52.787853 18611 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:08:52.790910 18627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:52.792237 18611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:08:52.792423 18611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "56c474eeae6b476cb09ef84973019f4d"
format_stamp: "Formatted at 2026-09-28 02:08:07 on dist-test-slave-23bp"
server_key: "964d2e5a817e0cef59f01aeb8641583f"
server_key_iv: "2d7dd619b74af3dc987f4e6cce4732c5"
server_key_version: "encryptionkey@0"
I20260928 02:08:52.793394 18611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:08:52.807884 18611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:08:52.812265 18611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:08:52.812551 18611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:08:52.813511 18611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:08:52.815227 18611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:08:52.815340 18611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:52.815447 18611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:08:52.815532 18611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:52.847910 18611 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:39855
I20260928 02:08:52.848006 18739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:39855 every 8 connection(s)
I20260928 02:08:52.849385 18611 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:08:52.850647 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 18611
I20260928 02:08:52.851500 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:45615
--local_ip_for_outbound_sockets=127.1.182.131
--tserver_master_addrs=127.1.182.190:39819
--webserver_port=35009
--webserver_interface=127.1.182.131
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:08:52.854951 18742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:52.852988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.130:55587)
0928 02:08:52.853798 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:52.854009 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":321,"threads_started":1}
W20260928 02:08:52.855727 18740 heartbeater.cc:644] Failed to heartbeat to 127.1.182.190:39819 (0 consecutive failures): Network error: Failed to ping master at 127.1.182.190:39819: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
I20260928 02:08:52.856607 18742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:52.856289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.130:45969)
0928 02:08:52.856433 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:52.856500 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 02:08:52.857483 18742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:52.857152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.130:49859)
0928 02:08:52.857292 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:52.857376 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:08:52.857714 18740 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:08:53.180222 18743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:08:53.180495 18743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:08:53.180667 18743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:08:53.187875 18743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:08:53.187979 18743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:08:53.188174 18743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:08:53.197999 18743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:45615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=35009
--tserver_master_addrs=127.1.182.190:39819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:08:53.199318 18743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:08:53.200654 18743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:08:53.219405 18748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:53.219482 18751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:08:53.219457 18749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:08:53.220480 18743 server_base.cc:1034] running on GCE node
I20260928 02:08:53.221113 18743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:08:53.222100 18743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:08:53.223295 18743 hybrid_clock.cc:648] HybridClock initialized: now 1790561333223247 us; error 39 us; skew 500 ppm
I20260928 02:08:53.223584 18743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:08:53 dist-test-slave-23bp krb5kdc[17282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561333, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:08:53.226900 18743 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:08:53.228466 18743 webserver.cc:460] Webserver started at http://127.1.182.131:35009/ using document root <none> and password file <none>
I20260928 02:08:53.229138 18743 fs_manager.cc:359] Metadata directory not provided
I20260928 02:08:53.229240 18743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:08:53.234624 18743 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:08:53.237354 18759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:53.238686 18743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:08:53.238863 18743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "d8e99f10ce32441db3ead9ed7901c894"
format_stamp: "Formatted at 2026-09-28 02:08:07 on dist-test-slave-23bp"
server_key: "e64e0f65a0038e1169946e29d6475f49"
server_key_iv: "a57fb90c42ee8918b5766c3a08954278"
server_key_version: "encryptionkey@0"
I20260928 02:08:53.239928 18743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:08:53.252015 18743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:08:53.256198 18743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:08:53.256464 18743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:08:53.257419 18743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:08:53.258746 18743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:08:53.258868 18743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:53.258993 18743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:08:53.259109 18743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:08:53.296811 18743 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:45615
I20260928 02:08:53.297067 18871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:45615 every 8 connection(s)
I20260928 02:08:53.299018 18743 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:08:53.304140 18874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:53.302216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.131:48977)
0928 02:08:53.302965 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:53.303272 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":286,"threads_started":1}
W20260928 02:08:53.304859 18872 heartbeater.cc:644] Failed to heartbeat to 127.1.182.190:39819 (0 consecutive failures): Network error: Failed to ping master at 127.1.182.190:39819: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
I20260928 02:08:53.305665 18874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:53.305361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.131:38073)
0928 02:08:53.305497 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:53.305562 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 02:08:53.306506 18874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:53.306221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.131:40517)
0928 02:08:53.306336 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:53.306404 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 02:08:53.306748 18872 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:08:53.307312 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 18743
I20260928 02:08:53.407670 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:53.406888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.129:57351)
0928 02:08:53.407398 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:53.407485 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":299,"threads_started":1}
I20260928 02:08:53.859169 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:53.858247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.130:34467)
0928 02:08:53.858774 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:53.858880 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":283,"threads_started":1}
I20260928 02:08:54.150550 18878 raft_consensus.cc:482] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:08:54.150846 18878 raft_consensus.cc:504] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:54.152396 18878 leader_election.cc:288] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8e99f10ce32441db3ead9ed7901c894 (127.1.182.131:45615), 56c474eeae6b476cb09ef84973019f4d (127.1.182.130:39855)
Sep 28 02:08:54 dist-test-slave-23bp krb5kdc[17282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561332, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.131@KRBTEST.COM
Sep 28 02:08:54 dist-test-slave-23bp krb5kdc[17282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561332, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.130@KRBTEST.COM
I20260928 02:08:54.190748 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:54.152972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:55491 (local address 127.1.182.131:45615)
0928 02:08:54.165606 (+ 12634us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:54.165625 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:54.166246 (+ 621us) server_negotiation.cc:407] Connection header received
0928 02:08:54.169153 (+ 2907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:54.169164 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:54.169695 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:54.170075 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:08:54.172672 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:54.174579 (+ 1907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:54.177599 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:54.178606 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:54.178794 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:54.183819 (+ 5025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:08:54.183827 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:08:54.183842 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:08:54.183880 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:08:54.186920 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:54.187861 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:54.187874 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:54.187880 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:54.188034 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:54.188500 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:54.188510 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:54.188512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:54.189004 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:08:54.189199 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:54.189961 (+ 762us) server_negotiation.cc:299] Negotiation successful
0928 02:08:54.190316 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12445,"thread_start_us":12342,"threads_started":1}
I20260928 02:08:54.190759 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:54.152782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:39855 (local address 127.1.182.129:50243)
0928 02:08:54.166042 (+ 13260us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:54.166068 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:08:54.166715 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:08:54.170267 (+ 3552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:08:54.170278 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:08:54.171250 (+ 972us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:08:54.172438 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:54.172467 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:54.174871 (+ 2404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:54.174877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:54.177435 (+ 2558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:54.177449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:54.178858 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:54.178865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:54.179213 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:54.180160 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:08:54.180199 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:08:54.183609 (+ 3410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:08:54.187177 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:08:54.187183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:08:54.187187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:08:54.187720 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:08:54.188169 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:08:54.188174 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:08:54.188178 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:08:54.188369 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:08:54.189162 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:08:54.189170 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:08:54.189474 (+ 304us) client_negotiation.cc:769] Sending connection context
0928 02:08:54.189758 (+ 284us) client_negotiation.cc:240] Negotiation successful
0928 02:08:54.190285 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13169,"mutex_wait_us":472,"spinlock_wait_cycles":54656,"thread_start_us":13038,"threads_started":1}
I20260928 02:08:54.190755 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:54.153030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:50243 (local address 127.1.182.130:39855)
0928 02:08:54.165375 (+ 12345us) server_negotiation.cc:206] Beginning negotiation
0928 02:08:54.165401 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 02:08:54.166245 (+ 844us) server_negotiation.cc:407] Connection header received
0928 02:08:54.169296 (+ 3051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:08:54.169305 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:08:54.169695 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:08:54.170075 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:08:54.172672 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:54.174579 (+ 1907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:54.177599 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:08:54.178605 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:08:54.178794 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:54.183897 (+ 5103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:08:54.183904 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:08:54.183918 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:08:54.183960 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:08:54.186985 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:54.187863 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:54.187879 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:54.187885 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:54.188040 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:08:54.188490 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:08:54.188495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:08:54.188498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:08:54.188999 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:08:54.189209 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 02:08:54.189983 (+ 774us) server_negotiation.cc:299] Negotiation successful
0928 02:08:54.190365 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12076,"thread_start_us":11855,"threads_started":1}
I20260928 02:08:54.191246 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:54.152752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:45615 (local address 127.1.182.129:55491)
0928 02:08:54.166011 (+ 13259us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:54.166069 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:08:54.166715 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:08:54.170906 (+ 4191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:08:54.170915 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:08:54.171510 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:08:54.172438 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:54.172467 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:54.174874 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:54.174879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:54.177433 (+ 2554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:08:54.177449 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:08:54.178850 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:08:54.178857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:08:54.179208 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:08:54.180160 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:08:54.180200 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:08:54.183505 (+ 3305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:08:54.187129 (+ 3624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:08:54.187140 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:08:54.187157 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:08:54.187714 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:08:54.188157 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:08:54.188163 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:08:54.188167 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:08:54.188379 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:08:54.189162 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:08:54.189170 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:08:54.189474 (+ 304us) client_negotiation.cc:769] Sending connection context
0928 02:08:54.189757 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 02:08:54.190437 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13095,"mutex_wait_us":613,"spinlock_wait_cycles":24576,"thread_start_us":12914,"threads_started":1}
W20260928 02:08:54.194458 18496 leader_election.cc:341] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56c474eeae6b476cb09ef84973019f4d (127.1.182.130:39855): Not found: Tablet not found: b3d43cfe06d9423c8b1b153b47a708c5
W20260928 02:08:54.194743 18495 leader_election.cc:341] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8e99f10ce32441db3ead9ed7901c894 (127.1.182.131:45615): Not found: Tablet not found: b3d43cfe06d9423c8b1b153b47a708c5
I20260928 02:08:54.194875 18495 leader_election.cc:302] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fba0fbc52616461c9445f8143f74d127; no voters: 56c474eeae6b476cb09ef84973019f4d, d8e99f10ce32441db3ead9ed7901c894
I20260928 02:08:54.195390 18878 raft_consensus.cc:2726] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:08:54.308321 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:54.307447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.131:39777)
0928 02:08:54.307917 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:54.308096 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":239,"threads_started":1}
I20260928 02:08:54.409039 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:54.408627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.129:56691)
0928 02:08:54.408800 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:54.408906 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 02:08:54.860955 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:54.860043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.130:54123)
0928 02:08:54.860615 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:54.860723 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":298,"threads_started":1}
I20260928 02:08:55.310062 18885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:55.309314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.131:59967)
0928 02:08:55.309767 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:55.309850 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":245,"threads_started":1}
I20260928 02:08:55.410720 18886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:55.409967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.129:50893)
0928 02:08:55.410453 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:55.410547 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":267,"threads_started":1}
I20260928 02:08:55.862500 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:55.861838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.130:58371)
0928 02:08:55.862256 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:55.862337 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":229,"threads_started":1}
I20260928 02:08:56.151352 18888 raft_consensus.cc:482] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:08:56.151551 18888 raft_consensus.cc:504] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:56.152156 18888 leader_election.cc:288] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8e99f10ce32441db3ead9ed7901c894 (127.1.182.131:45615), 56c474eeae6b476cb09ef84973019f4d (127.1.182.130:39855)
W20260928 02:08:56.153071 18495 leader_election.cc:341] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8e99f10ce32441db3ead9ed7901c894 (127.1.182.131:45615): Not found: Tablet not found: b3d43cfe06d9423c8b1b153b47a708c5
W20260928 02:08:56.153239 18496 leader_election.cc:341] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56c474eeae6b476cb09ef84973019f4d (127.1.182.130:39855): Not found: Tablet not found: b3d43cfe06d9423c8b1b153b47a708c5
I20260928 02:08:56.153424 18496 leader_election.cc:302] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fba0fbc52616461c9445f8143f74d127; no voters: 56c474eeae6b476cb09ef84973019f4d, d8e99f10ce32441db3ead9ed7901c894
I20260928 02:08:56.153663 18888 raft_consensus.cc:2726] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:08:56.311648 18889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:56.310878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.131:51637)
0928 02:08:56.311415 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:56.311492 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":338,"threads_started":1}
I20260928 02:08:56.412356 18890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:56.411615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.129:58971)
0928 02:08:56.412068 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:56.412174 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":211,"threads_started":1}
I20260928 02:08:56.864221 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:56.863439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.130:33141)
0928 02:08:56.863968 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:56.864045 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":319,"threads_started":1}
I20260928 02:08:57.313206 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:57.312481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.131:51703)
0928 02:08:57.312942 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:57.313028 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":233,"threads_started":1}
I20260928 02:08:57.414134 18893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:57.413362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.129:49789)
0928 02:08:57.413862 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:57.413943 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":234,"threads_started":1}
I20260928 02:08:57.866075 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:57.865220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.130:58845)
0928 02:08:57.865771 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:57.865868 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":273,"threads_started":1}
I20260928 02:08:58.314800 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:58.314115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.131:47503)
0928 02:08:58.314557 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:58.314642 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":230,"threads_started":1}
I20260928 02:08:58.415902 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:58.415198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.129:40223)
0928 02:08:58.415664 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:58.415742 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":253,"threads_started":1}
I20260928 02:08:58.868136 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:58.867155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.130:35455)
0928 02:08:58.867783 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:58.867894 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":318,"threads_started":1}
I20260928 02:08:59.085642 18898 raft_consensus.cc:482] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:08:59.085865 18898 raft_consensus.cc:504] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:08:59.086591 18898 leader_election.cc:288] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8e99f10ce32441db3ead9ed7901c894 (127.1.182.131:45615), 56c474eeae6b476cb09ef84973019f4d (127.1.182.130:39855)
W20260928 02:08:59.087513 18495 leader_election.cc:341] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8e99f10ce32441db3ead9ed7901c894 (127.1.182.131:45615): Not found: Tablet not found: b3d43cfe06d9423c8b1b153b47a708c5
W20260928 02:08:59.087713 18496 leader_election.cc:341] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56c474eeae6b476cb09ef84973019f4d (127.1.182.130:39855): Not found: Tablet not found: b3d43cfe06d9423c8b1b153b47a708c5
I20260928 02:08:59.087868 18496 leader_election.cc:302] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fba0fbc52616461c9445f8143f74d127; no voters: 56c474eeae6b476cb09ef84973019f4d, d8e99f10ce32441db3ead9ed7901c894
I20260928 02:08:59.088121 18898 raft_consensus.cc:2726] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:08:59.317025 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:59.316216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.131:42031)
0928 02:08:59.316765 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:59.316845 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":323,"threads_started":1}
I20260928 02:08:59.417917 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:59.417109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.129:33119)
0928 02:08:59.417610 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:59.417718 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":253,"threads_started":1}
I20260928 02:08:59.870112 18901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:08:59.869183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.130:47971)
0928 02:08:59.869737 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0928 02:08:59.869871 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":248,"threads_started":1}
I20260928 02:09:00.318734 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:00.317949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.131:57965)
0928 02:09:00.318455 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:00.318538 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":237,"threads_started":1}
I20260928 02:09:00.419665 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:00.418839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.129:39411)
0928 02:09:00.419384 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:00.419482 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":293,"threads_started":1}
I20260928 02:09:00.871821 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:00.871136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.130:38865)
0928 02:09:00.871560 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:00.871649 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":211,"threads_started":1}
I20260928 02:09:01.320394 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:01.319688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.131:51099)
0928 02:09:01.320118 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:01.320206 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":218,"threads_started":1}
I20260928 02:09:01.421377 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:01.420600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.129:50553)
0928 02:09:01.421103 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:01.421191 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":230,"threads_started":1}
I20260928 02:09:01.873755 18907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:01.872810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.130:58777)
0928 02:09:01.873454 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:01.873578 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":331,"threads_started":1}
I20260928 02:09:02.069271 18908 raft_consensus.cc:482] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:09:02.069502 18908 raft_consensus.cc:504] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:09:02.070227 18908 leader_election.cc:288] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8e99f10ce32441db3ead9ed7901c894 (127.1.182.131:45615), 56c474eeae6b476cb09ef84973019f4d (127.1.182.130:39855)
W20260928 02:09:02.071328 18495 leader_election.cc:341] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8e99f10ce32441db3ead9ed7901c894 (127.1.182.131:45615): Not found: Tablet not found: b3d43cfe06d9423c8b1b153b47a708c5
W20260928 02:09:02.071512 18496 leader_election.cc:341] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56c474eeae6b476cb09ef84973019f4d (127.1.182.130:39855): Not found: Tablet not found: b3d43cfe06d9423c8b1b153b47a708c5
I20260928 02:09:02.071630 18496 leader_election.cc:302] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fba0fbc52616461c9445f8143f74d127; no voters: 56c474eeae6b476cb09ef84973019f4d, d8e99f10ce32441db3ead9ed7901c894
I20260928 02:09:02.071852 18908 raft_consensus.cc:2726] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:09:02.322144 18909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:02.321350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.131:47949)
0928 02:09:02.321863 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:02.321941 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":272,"threads_started":1}
I20260928 02:09:02.423292 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:02.422431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.129:51705)
0928 02:09:02.422894 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:02.423056 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":220,"threads_started":1}
I20260928 02:09:02.875625 18911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:02.874741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.130:38355)
0928 02:09:02.875290 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:02.875432 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":253,"threads_started":1}
I20260928 02:09:03.324064 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:03.323154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.131:36371)
0928 02:09:03.323663 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:03.323771 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":246,"threads_started":1}
I20260928 02:09:03.425205 18913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:03.424341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.129:44505)
0928 02:09:03.424907 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:03.425015 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":323,"threads_started":1}
I20260928 02:09:03.877276 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:03.876539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.130:33351)
0928 02:09:03.877026 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:03.877111 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":221,"threads_started":1}
I20260928 02:09:04.325690 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:04.324954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.131:45603)
0928 02:09:04.325418 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:04.325529 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":238,"threads_started":1}
I20260928 02:09:04.426939 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:04.426197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.129:52597)
0928 02:09:04.426682 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:04.426773 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":233,"threads_started":1}
I20260928 02:09:04.879424 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:04.878404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.130:42349)
0928 02:09:04.879029 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:04.879191 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":316,"threads_started":1}
I20260928 02:09:05.327337 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:05.326586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.131:48707)
0928 02:09:05.327034 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:05.327164 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":213,"threads_started":1}
I20260928 02:09:05.428553 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:05.427809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.129:37445)
0928 02:09:05.428303 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:05.428381 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":297,"threads_started":1}
I20260928 02:09:05.821810 18920 raft_consensus.cc:482] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:09:05.822006 18920 raft_consensus.cc:504] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fba0fbc52616461c9445f8143f74d127" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 37921 } } peers { permanent_uuid: "d8e99f10ce32441db3ead9ed7901c894" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 45615 } } peers { permanent_uuid: "56c474eeae6b476cb09ef84973019f4d" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39855 } }
I20260928 02:09:05.822782 18920 leader_election.cc:288] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8e99f10ce32441db3ead9ed7901c894 (127.1.182.131:45615), 56c474eeae6b476cb09ef84973019f4d (127.1.182.130:39855)
W20260928 02:09:05.823851 18495 leader_election.cc:341] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8e99f10ce32441db3ead9ed7901c894 (127.1.182.131:45615): Not found: Tablet not found: b3d43cfe06d9423c8b1b153b47a708c5
W20260928 02:09:05.824038 18496 leader_election.cc:341] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 56c474eeae6b476cb09ef84973019f4d (127.1.182.130:39855): Not found: Tablet not found: b3d43cfe06d9423c8b1b153b47a708c5
I20260928 02:09:05.824213 18496 leader_election.cc:302] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fba0fbc52616461c9445f8143f74d127; no voters: 56c474eeae6b476cb09ef84973019f4d, d8e99f10ce32441db3ead9ed7901c894
I20260928 02:09:05.824538 18920 raft_consensus.cc:2726] T b3d43cfe06d9423c8b1b153b47a708c5 P fba0fbc52616461c9445f8143f74d127 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:09:05.881458 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:05.880531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.130:53349)
0928 02:09:05.881115 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:05.881211 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":301,"threads_started":1}
I20260928 02:09:06.329136 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:06.328297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.131:37677)
0928 02:09:06.328842 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:06.328938 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":284,"threads_started":1}
I20260928 02:09:06.430142 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:06.429396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.129:48661)
0928 02:09:06.429873 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:06.429969 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":244,"threads_started":1}
I20260928 02:09:06.883307 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:06.882435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.130:57685)
0928 02:09:06.882924 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:06.883032 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":257,"threads_started":1}
I20260928 02:09:07.330789 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:07.330022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.131:36257)
0928 02:09:07.330496 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:07.330589 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":234,"threads_started":1}
I20260928 02:09:07.432016 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:07.431299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.129:36829)
0928 02:09:07.431760 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:07.431844 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":228,"threads_started":1}
I20260928 02:09:07.885783 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:07.884850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.130:36361)
0928 02:09:07.885422 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:07.885587 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":269,"threads_started":1}
I20260928 02:09:08.308923 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 18436
I20260928 02:09:08.331713 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:08.310104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51320 (local address 127.1.182.129:37921)
0928 02:09:08.310684 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:08.310696 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:08.310980 (+ 284us) server_negotiation.cc:407] Connection header received
0928 02:09:08.311232 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:08.311240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:08.311508 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:08.311651 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:08.313107 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:08.314066 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:08.315352 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:08.315888 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:08.315997 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:08.325369 (+ 9372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:08.325382 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:08.325398 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:08.325442 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:08.328678 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:08.329386 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:08.329393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:08.329395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:08.329537 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:08.329967 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:08.329971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:08.329973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:08.330362 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:08.330726 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:08.331043 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 02:09:08.331356 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":226,"threads_started":1}
I20260928 02:09:08.332894 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:08.331993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:39819 (local address 127.1.182.131:35621)
0928 02:09:08.332569 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:08.332700 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.190:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":262,"threads_started":1}
I20260928 02:09:08.333072 18603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:08.427357 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 18436
I20260928 02:09:08.446230 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 18611
I20260928 02:09:08.460047 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:08.447086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51574 (local address 127.1.182.130:39855)
0928 02:09:08.447635 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:08.447640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:08.447657 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:09:08.447714 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:08.447718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:08.447776 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:08.447869 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:08.449188 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:08.449981 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:08.451431 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:08.451838 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:08.451932 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:08.454506 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:08.454512 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:08.454517 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:08.454549 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:08.457300 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:08.457930 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:08.457934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:08.457936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:08.458102 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:08.458475 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:08.458478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:08.458480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:08.458762 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:08.458971 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:08.459439 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 02:09:08.459620 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":314,"threads_started":1}
I20260928 02:09:08.460801 18735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:08.546808 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 18611
I20260928 02:09:08.564570 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 18743
I20260928 02:09:08.578374 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:08.565401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50690 (local address 127.1.182.131:45615)
0928 02:09:08.565961 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:08.565966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:08.565981 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:09:08.566035 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:08.566039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:08.566105 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:08.566227 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:08.567643 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:08.568415 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:08.569666 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:08.570103 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:08.570239 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:08.572981 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:08.572986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:08.572989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:08.573021 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:08.575671 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:08.576364 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:08.576375 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:08.576378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:08.576503 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:08.576879 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:08.576885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:08.576888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:08.577162 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:08.577449 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:08.577740 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 02:09:08.577973 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":265,"threads_started":1}
I20260928 02:09:08.579124 18867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:08.663606 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 18743
2026-09-28T02:09:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64240 ms)
[----------] 30 tests from SecurityITest (295899 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:09:08 dist-test-slave-23bp krb5kdc[18937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:09:08 dist-test-slave-23bp krb5kdc[18937](info): set up 2 sockets
Sep 28 02:09:08 dist-test-slave-23bp krb5kdc[18937](info): commencing operation
krb5kdc: starting...
W20260928 02:09:09.837621 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:09:09 dist-test-slave-23bp krb5kdc[18937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:09:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:09:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:09:10.097265 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:43965
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:44479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:43965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:10.433213 18950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:10.433506 18950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:10.433660 18950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:10.440953 18950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:09:10.441064 18950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:10.441162 18950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:10.441256 18950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:09:10.441331 18950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:09:10.451655 18950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:44479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:43965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:43965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:10.453099 18950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:10.454515 18950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:10.466886 18956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:10.466847 18958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:10.466918 18955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:10.467399 18950 server_base.cc:1034] running on GCE node
I20260928 02:09:10.468365 18950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:10.469833 18950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:10.471192 18950 hybrid_clock.cc:648] HybridClock initialized: now 1790561350471131 us; error 55 us; skew 500 ppm
I20260928 02:09:10.471571 18950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:10 dist-test-slave-23bp krb5kdc[18937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561350, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:10.474869 18950 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:09:10.476588 18950 webserver.cc:460] Webserver started at http://127.1.182.190:38069/ using document root <none> and password file <none>
I20260928 02:09:10.477319 18950 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:10.477464 18950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:10.477785 18950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:10.480937 18950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "121ee6244c9345a792acb98aa8de6cd5"
format_stamp: "Formatted at 2026-09-28 02:09:10 on dist-test-slave-23bp"
server_key: "78fd6ec10847c8572eafcb23765060d9"
server_key_iv: "268bf8412c6083fe18a32352557c1412"
server_key_version: "encryptionkey@0"
I20260928 02:09:10.481709 18950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "121ee6244c9345a792acb98aa8de6cd5"
format_stamp: "Formatted at 2026-09-28 02:09:10 on dist-test-slave-23bp"
server_key: "78fd6ec10847c8572eafcb23765060d9"
server_key_iv: "268bf8412c6083fe18a32352557c1412"
server_key_version: "encryptionkey@0"
I20260928 02:09:10.486908 18950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:09:10.490613 18965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:10.492187 18950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:09:10.492329 18950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "121ee6244c9345a792acb98aa8de6cd5"
format_stamp: "Formatted at 2026-09-28 02:09:10 on dist-test-slave-23bp"
server_key: "78fd6ec10847c8572eafcb23765060d9"
server_key_iv: "268bf8412c6083fe18a32352557c1412"
server_key_version: "encryptionkey@0"
I20260928 02:09:10.492508 18950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:10.522984 18950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:10.527097 18950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:10.527344 18950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:10.546440 18950 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:43965
I20260928 02:09:10.546458 19017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:43965 every 8 connection(s)
I20260928 02:09:10.547926 18950 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:09:10.551877 19018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:10.552348 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 18950
I20260928 02:09:10.552557 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:09:10.553211 1754 external_mini_cluster.cc:1428] Setting key 52d744eb226de27d0485e1095c7a4af3
I20260928 02:09:10.564358 19018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5: Bootstrap starting.
Sep 28 02:09:10 dist-test-slave-23bp krb5kdc[18937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:09:10.567682 19018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:10.568735 19018 log.cc:824] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:10.572072 19018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5: No bootstrap required, opened a new log
I20260928 02:09:10.576301 19021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:10.555373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41512 (local address 127.1.182.190:43965)
0928 02:09:10.556159 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:10.556171 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:10.556216 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:09:10.558091 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:10.558117 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:10.558554 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:10.558981 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:10.560587 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:10.561920 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:10.563374 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:10.564064 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:10.564261 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:10.568543 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:10.568551 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:10.568577 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:10.568617 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:10.572038 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:10.572774 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:10.572781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:10.572787 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:10.572944 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:10.573354 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:10.573358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:10.573360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:10.573915 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:10.574255 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:10.574707 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 02:09:10.575234 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":328,"threads_started":1}
I20260928 02:09:10.580583 19018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121ee6244c9345a792acb98aa8de6cd5" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 43965 } }
I20260928 02:09:10.580966 19018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:10.581073 19018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121ee6244c9345a792acb98aa8de6cd5, State: Initialized, Role: FOLLOWER
I20260928 02:09:10.581661 19018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121ee6244c9345a792acb98aa8de6cd5" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 43965 } }
I20260928 02:09:10.581907 19018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:09:10.582067 19018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:09:10.582231 19018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:10.584474 19018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121ee6244c9345a792acb98aa8de6cd5" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 43965 } }
I20260928 02:09:10.584969 19018 leader_election.cc:302] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 121ee6244c9345a792acb98aa8de6cd5; no voters:
I20260928 02:09:10.585665 19018 leader_election.cc:288] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:09:10.585803 19023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:10.586549 19023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5 [term 1 LEADER]: Becoming Leader. State: Replica: 121ee6244c9345a792acb98aa8de6cd5, State: Running, Role: LEADER
I20260928 02:09:10.587201 19023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121ee6244c9345a792acb98aa8de6cd5" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 43965 } }
I20260928 02:09:10.588080 19018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:09:10.590292 19024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "121ee6244c9345a792acb98aa8de6cd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121ee6244c9345a792acb98aa8de6cd5" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 43965 } } }
I20260928 02:09:10.590674 19024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:10.591285 19025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 121ee6244c9345a792acb98aa8de6cd5. Latest consensus state: current_term: 1 leader_uuid: "121ee6244c9345a792acb98aa8de6cd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121ee6244c9345a792acb98aa8de6cd5" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 43965 } } }
I20260928 02:09:10.591740 19025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:10.594135 19028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:09:10.598959 19028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:09:10.608675 19028 catalog_manager.cc:1347] Generated new cluster ID: d4db03cdf54f43aa86903fd9de62cfce
I20260928 02:09:10.608794 19028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:09:10.630867 19028 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:09:10.632839 19028 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:09:10.647917 19028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5: Generated new TSK 0
I20260928 02:09:10.648823 19028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.182.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
I20260928 02:09:10.712260 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:43965
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:44479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:11.046793 19044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:11.047152 19044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:11.047264 19044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:11.054587 19044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:11.054705 19044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:11.054823 19044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:09:11.064837 19044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:44479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:43965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:11.066113 19044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:11.067444 19044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:11.082062 19050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:11.082150 19049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:11.082127 19052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:11.082943 19044 server_base.cc:1034] running on GCE node
I20260928 02:09:11.083576 19044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:11.084506 19044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:11.085701 19044 hybrid_clock.cc:648] HybridClock initialized: now 1790561351085634 us; error 62 us; skew 500 ppm
I20260928 02:09:11.085980 19044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:11 dist-test-slave-23bp krb5kdc[18937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561351, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:11.089162 19044 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:09:11.090749 19044 webserver.cc:460] Webserver started at http://127.1.182.129:40869/ using document root <none> and password file <none>
I20260928 02:09:11.091482 19044 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:11.091593 19044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:11.091856 19044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:11.095099 19044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "765011ee0b9b4156b75513a6144d93ac"
format_stamp: "Formatted at 2026-09-28 02:09:11 on dist-test-slave-23bp"
server_key: "df4806856d046fabe7598f0f79153d87"
server_key_iv: "53672a38a5c1e87020127a4002485838"
server_key_version: "encryptionkey@0"
I20260928 02:09:11.095959 19044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "765011ee0b9b4156b75513a6144d93ac"
format_stamp: "Formatted at 2026-09-28 02:09:11 on dist-test-slave-23bp"
server_key: "df4806856d046fabe7598f0f79153d87"
server_key_iv: "53672a38a5c1e87020127a4002485838"
server_key_version: "encryptionkey@0"
I20260928 02:09:11.101270 19044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:09:11.104782 19059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:11.106132 19044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:09:11.106297 19044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "765011ee0b9b4156b75513a6144d93ac"
format_stamp: "Formatted at 2026-09-28 02:09:11 on dist-test-slave-23bp"
server_key: "df4806856d046fabe7598f0f79153d87"
server_key_iv: "53672a38a5c1e87020127a4002485838"
server_key_version: "encryptionkey@0"
I20260928 02:09:11.106492 19044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:11.121783 19044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:11.125881 19044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:11.126180 19044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:11.127209 19044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:09:11.128547 19044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:09:11.128665 19044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:11.128826 19044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:09:11.128890 19044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:11.164496 19044 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:40417
I20260928 02:09:11.164587 19172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:40417 every 8 connection(s)
I20260928 02:09:11.165818 19044 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:09:11.166806 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 19044
I20260928 02:09:11.167004 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:09:11.167785 1754 external_mini_cluster.cc:1428] Setting key f5622caf472e4581cd73a525533f17ad
Sep 28 02:09:11 dist-test-slave-23bp krb5kdc[18937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561351, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:09:11.189819 19175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:11.168615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:48817 (local address 127.1.182.190:43965)
0928 02:09:11.169123 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:11.169129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:11.170259 (+ 1130us) server_negotiation.cc:407] Connection header received
0928 02:09:11.173220 (+ 2961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:11.173224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:11.173290 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:11.173384 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:11.175534 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:11.176314 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:11.178877 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:11.179381 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:11.179581 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:11.183435 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:11.183440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:11.183443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:11.183477 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:11.186013 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:11.186994 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:11.187002 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:11.187008 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:11.187245 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:11.187912 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:11.187922 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:11.187926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:11.188215 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:11.188376 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:11.189208 (+ 832us) server_negotiation.cc:299] Negotiation successful
0928 02:09:11.189442 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":785,"threads_started":1}
I20260928 02:09:11.190884 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:11.169250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:43965 (local address 127.1.182.129:48817)
0928 02:09:11.170076 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:11.170143 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:11.172909 (+ 2766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:11.173658 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:11.173671 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:11.174462 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:09:11.175342 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:11.175369 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:11.176490 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:11.176498 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:11.178673 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:11.178691 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:11.179629 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:11.179634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:11.179835 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:11.180622 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:09:11.180655 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:09:11.183213 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:09:11.186156 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:09:11.186165 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:09:11.186182 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:09:11.186807 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:09:11.187420 (+ 613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:09:11.187425 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:09:11.187429 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:09:11.187759 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:09:11.188350 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:09:11.188358 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:09:11.188727 (+ 369us) client_negotiation.cc:769] Sending connection context
0928 02:09:11.189073 (+ 346us) client_negotiation.cc:240] Negotiation successful
0928 02:09:11.189633 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":316,"threads_started":1}
I20260928 02:09:11.192701 19173 heartbeater.cc:347] Connected to a master server at 127.1.182.190:43965
I20260928 02:09:11.193017 19173 heartbeater.cc:464] Registering TS with master...
I20260928 02:09:11.193769 19173 heartbeater.cc:511] Master 127.1.182.190:43965 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.182.130@KRBTEST.COM; defaulting to no policy
I20260928 02:09:11.196097 18982 ts_manager.cc:194] Registered new tserver with Master: 765011ee0b9b4156b75513a6144d93ac (127.1.182.129:40417)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.130@KRBTEST.COM" created.
I20260928 02:09:11.197877 18982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:48817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
I20260928 02:09:11.229888 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:43965
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:44479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:11.584717 19179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:11.585016 19179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:11.585201 19179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:11.592787 19179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:11.592937 19179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:11.593135 19179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:09:11.603873 19179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:44479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:43965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:11.605290 19179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:11.606645 19179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:11.620337 19184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:11.620363 19185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:11.620343 19187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:11.621625 19179 server_base.cc:1034] running on GCE node
I20260928 02:09:11.622290 19179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:11.623349 19179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:11.624606 19179 hybrid_clock.cc:648] HybridClock initialized: now 1790561351624557 us; error 43 us; skew 500 ppm
I20260928 02:09:11.624957 19179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:11 dist-test-slave-23bp krb5kdc[18937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561351, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:11.628355 19179 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:09:11.629956 19179 webserver.cc:460] Webserver started at http://127.1.182.130:39585/ using document root <none> and password file <none>
I20260928 02:09:11.630698 19179 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:11.630851 19179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:11.631248 19179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:11.634292 19179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "d57f0e88f92b44038517c42f0cff68af"
format_stamp: "Formatted at 2026-09-28 02:09:11 on dist-test-slave-23bp"
server_key: "f274a1615c9839ff2be85aa7bc836e86"
server_key_iv: "646def5c7506ea54974c18e6091f6d11"
server_key_version: "encryptionkey@0"
I20260928 02:09:11.635259 19179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "d57f0e88f92b44038517c42f0cff68af"
format_stamp: "Formatted at 2026-09-28 02:09:11 on dist-test-slave-23bp"
server_key: "f274a1615c9839ff2be85aa7bc836e86"
server_key_iv: "646def5c7506ea54974c18e6091f6d11"
server_key_version: "encryptionkey@0"
I20260928 02:09:11.640496 19179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 02:09:11.644008 19194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:11.645291 19179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:09:11.645471 19179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "d57f0e88f92b44038517c42f0cff68af"
format_stamp: "Formatted at 2026-09-28 02:09:11 on dist-test-slave-23bp"
server_key: "f274a1615c9839ff2be85aa7bc836e86"
server_key_iv: "646def5c7506ea54974c18e6091f6d11"
server_key_version: "encryptionkey@0"
I20260928 02:09:11.645687 19179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:11.664805 19179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:11.669476 19179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:11.669796 19179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:11.670833 19179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:09:11.672278 19179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:09:11.672384 19179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:11.672538 19179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:09:11.672610 19179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:11.707288 19179 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:37603
I20260928 02:09:11.707333 19308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:37603 every 8 connection(s)
I20260928 02:09:11.708609 19179 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:09:11.716501 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 19179
I20260928 02:09:11.716708 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:09:11.717198 1754 external_mini_cluster.cc:1428] Setting key d85e8b4b76b213d501c2708d96a944ac
Sep 28 02:09:11 dist-test-slave-23bp krb5kdc[18937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561351, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:09:11.730227 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:11.711209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:59291 (local address 127.1.182.190:43965)
0928 02:09:11.711665 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:11.711670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:11.712729 (+ 1059us) server_negotiation.cc:407] Connection header received
0928 02:09:11.714613 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:11.714619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:11.714711 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:11.714829 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:11.716554 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:11.717407 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:11.719376 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:11.719795 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:11.719890 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:11.724131 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:11.724136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:11.724139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:11.724170 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:11.726742 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:11.727675 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:11.727682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:11.727686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:11.727934 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:11.728478 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:11.728483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:11.728486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:11.728745 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:11.728904 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:11.729619 (+ 715us) server_negotiation.cc:299] Negotiation successful
0928 02:09:11.729849 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":241,"threads_started":1}
I20260928 02:09:11.731293 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:11.711873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:43965 (local address 127.1.182.130:59291)
0928 02:09:11.712571 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:11.712609 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:11.714349 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:11.715136 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:11.715145 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:11.715660 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:09:11.716357 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:11.716385 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:11.717559 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:11.717573 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:11.719234 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:11.719252 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:11.720029 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:11.720039 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:11.720365 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:11.721109 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:09:11.721151 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:09:11.723847 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:09:11.726888 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:09:11.726897 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:09:11.726913 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:09:11.727492 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:09:11.728060 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:09:11.728065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:09:11.728068 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:09:11.728352 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:09:11.728876 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:09:11.728884 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:09:11.729233 (+ 349us) client_negotiation.cc:769] Sending connection context
0928 02:09:11.729495 (+ 262us) client_negotiation.cc:240] Negotiation successful
0928 02:09:11.730036 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":275,"threads_started":1}
I20260928 02:09:11.732739 19309 heartbeater.cc:347] Connected to a master server at 127.1.182.190:43965
I20260928 02:09:11.733069 19309 heartbeater.cc:464] Registering TS with master...
I20260928 02:09:11.733911 19309 heartbeater.cc:511] Master 127.1.182.190:43965 requested a full tablet report, sending...
I20260928 02:09:11.735606 18982 ts_manager.cc:194] Registered new tserver with Master: d57f0e88f92b44038517c42f0cff68af (127.1.182.130:37603)
I20260928 02:09:11.736861 18982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:59291
WARNING: no policy specified for kudu/127.1.182.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
I20260928 02:09:11.774600 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:43965
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:44479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:12.107402 19315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:12.107681 19315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:12.107775 19315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:12.115774 19315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:12.115893 19315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:12.116019 19315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:09:12.127856 19315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:44479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:43965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:12.129390 19315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:12.131065 19315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:12.144932 19320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:12.144891 19321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:12.145826 19323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:12.146590 19315 server_base.cc:1034] running on GCE node
I20260928 02:09:12.147264 19315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:12.148209 19315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:12.149444 19315 hybrid_clock.cc:648] HybridClock initialized: now 1790561352149388 us; error 68 us; skew 500 ppm
I20260928 02:09:12.149857 19315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:12 dist-test-slave-23bp krb5kdc[18937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561352, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:12.153210 19315 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:09:12.154685 19315 webserver.cc:460] Webserver started at http://127.1.182.131:37195/ using document root <none> and password file <none>
I20260928 02:09:12.155484 19315 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:12.155632 19315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:12.155941 19315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:12.158900 19315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "105d8ccb11db42f2b61c9ab2d111cf70"
format_stamp: "Formatted at 2026-09-28 02:09:12 on dist-test-slave-23bp"
server_key: "7db98359edcda0b3a510d9d6df23835a"
server_key_iv: "41288cae5a91ca91609bb904a502a861"
server_key_version: "encryptionkey@0"
I20260928 02:09:12.159803 19315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "105d8ccb11db42f2b61c9ab2d111cf70"
format_stamp: "Formatted at 2026-09-28 02:09:12 on dist-test-slave-23bp"
server_key: "7db98359edcda0b3a510d9d6df23835a"
server_key_iv: "41288cae5a91ca91609bb904a502a861"
server_key_version: "encryptionkey@0"
I20260928 02:09:12.165439 19315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260928 02:09:12.169872 19330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:12.171339 19315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:09:12.171480 19315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "105d8ccb11db42f2b61c9ab2d111cf70"
format_stamp: "Formatted at 2026-09-28 02:09:12 on dist-test-slave-23bp"
server_key: "7db98359edcda0b3a510d9d6df23835a"
server_key_iv: "41288cae5a91ca91609bb904a502a861"
server_key_version: "encryptionkey@0"
I20260928 02:09:12.171650 19315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:12.191874 19315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:12.196458 19315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:12.196796 19315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:12.197891 19315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:09:12.199441 19315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:09:12.199553 19315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:12.199713 19315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:09:12.199783 19315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:12.201756 19173 heartbeater.cc:503] Master 127.1.182.190:43965 was elected leader, sending a full tablet report...
I20260928 02:09:12.233932 19315 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:37273
I20260928 02:09:12.233978 19443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:37273 every 8 connection(s)
I20260928 02:09:12.235387 19315 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:09:12.239326 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 19315
I20260928 02:09:12.239524 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:09:12.240095 1754 external_mini_cluster.cc:1428] Setting key 5793a973c7e78a998f3af3fcf509a970
Sep 28 02:09:12 dist-test-slave-23bp krb5kdc[18937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561352, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:09:12.256487 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:12.237885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:59639 (local address 127.1.182.190:43965)
0928 02:09:12.238383 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:12.238388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:12.239325 (+ 937us) server_negotiation.cc:407] Connection header received
0928 02:09:12.241615 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:12.241628 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:12.241696 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:12.241789 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:12.243797 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.244606 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.246122 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.246551 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.246656 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:12.250145 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:12.250150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:12.250153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:12.250187 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:12.252871 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:12.253631 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:12.253635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:12.253638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:12.253781 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:12.254487 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:12.254490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:12.254493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:12.254844 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:12.254991 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:12.255969 (+ 978us) server_negotiation.cc:299] Negotiation successful
0928 02:09:12.256220 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":263,"threads_started":1}
I20260928 02:09:12.257454 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:12.238333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:43965 (local address 127.1.182.131:59639)
0928 02:09:12.239127 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:12.239190 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:12.241361 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:12.242095 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:12.242104 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:12.242815 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:09:12.243639 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:12.243660 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.244730 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.244734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:12.245962 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:12.245973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.246742 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.246746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:12.246939 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:12.247619 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:09:12.247645 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:09:12.249891 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:09:12.253016 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:09:12.253025 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:09:12.253039 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:09:12.253478 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:09:12.253903 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:09:12.253908 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:09:12.253911 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:09:12.254141 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:09:12.255000 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:09:12.255008 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:09:12.255527 (+ 519us) client_negotiation.cc:769] Sending connection context
0928 02:09:12.255818 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 02:09:12.256344 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":261,"threads_started":1}
I20260928 02:09:12.258919 19444 heartbeater.cc:347] Connected to a master server at 127.1.182.190:43965
I20260928 02:09:12.259250 19444 heartbeater.cc:464] Registering TS with master...
I20260928 02:09:12.259963 19444 heartbeater.cc:511] Master 127.1.182.190:43965 requested a full tablet report, sending...
I20260928 02:09:12.261332 18982 ts_manager.cc:194] Registered new tserver with Master: 105d8ccb11db42f2b61c9ab2d111cf70 (127.1.182.131:37273)
I20260928 02:09:12.262146 18982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:59639
I20260928 02:09:12.268481 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:09:12.285825 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:12.273134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41522 (local address 127.1.182.190:43965)
0928 02:09:12.273314 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:12.273320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:12.273794 (+ 474us) server_negotiation.cc:407] Connection header received
0928 02:09:12.273939 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:12.273943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:12.274008 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:12.274096 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:12.275796 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.276681 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.278014 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.278436 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.278546 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:12.280960 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:12.280965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:12.280967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:12.280998 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:12.283604 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:12.284199 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:12.284203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:12.284205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:12.284313 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:12.284625 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:12.284628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:12.284631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:12.284935 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:12.285113 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:12.285432 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 02:09:12.285606 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:09:12.290074 18982 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:09:12.297333 18982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:09:12.326011 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:12.310072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48906 (local address 127.1.182.131:37273)
0928 02:09:12.316615 (+ 6543us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:12.316629 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:12.316668 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:09:12.316776 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:12.316784 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:12.317097 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:12.318897 (+ 1800us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:12.319666 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.320911 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.323335 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.324695 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.324837 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:12.325073 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:12.325456 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 02:09:12.325643 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6309,"thread_start_us":6119,"threads_started":1}
I20260928 02:09:12.326004 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:12.309992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:40417 (local address 127.0.0.1:33748)
0928 02:09:12.316979 (+ 6987us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:12.317006 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:12.317161 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:12.318477 (+ 1316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:12.318486 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:12.318543 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:12.319018 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:12.319035 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.320997 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.321005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:12.323392 (+ 2387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:12.323406 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.325087 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.325095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:12.325230 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:12.325254 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:09:12.325340 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:09:12.325561 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6824,"mutex_wait_us":44,"spinlock_wait_cycles":28032,"thread_start_us":1197,"threads_started":1}
I20260928 02:09:12.326709 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:12.310555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33748 (local address 127.1.182.129:40417)
0928 02:09:12.317794 (+ 7239us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:12.317805 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:12.317836 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:09:12.317937 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:12.317944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:12.318242 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:12.319176 (+ 934us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:12.319441 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.320858 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.323585 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.324816 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.325472 (+ 656us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:12.325709 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:12.325794 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 02:09:12.325958 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7053,"thread_start_us":6858,"threads_started":1}
I20260928 02:09:12.326859 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:12.309724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:37273 (local address 127.0.0.1:48906)
0928 02:09:12.310896 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:12.310957 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:12.311211 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:12.318478 (+ 7267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:12.318485 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:12.318553 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:12.319455 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:12.319470 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.321091 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.321096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:12.323126 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:12.323145 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.324895 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.324903 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:12.325166 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:12.325226 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 02:09:12.325356 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 02:09:12.326424 (+ 1068us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":376,"spinlock_wait_cycles":3328,"thread_start_us":6904,"threads_started":1}
I20260928 02:09:12.327083 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:12.317943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:37603 (local address 127.0.0.1:46654)
0928 02:09:12.318453 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:12.318475 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:12.318702 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:12.319624 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:12.319628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:12.319653 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:12.320026 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:12.320036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.321893 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.321898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:12.324161 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:12.324175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.326171 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.326179 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:12.326354 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:12.326386 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:09:12.326521 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 02:09:12.326655 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"spinlock_wait_cycles":3968,"thread_start_us":254,"threads_started":1}
I20260928 02:09:12.327128 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:12.317300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46654 (local address 127.1.182.130:37603)
0928 02:09:12.317883 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:12.317890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:12.318558 (+ 668us) server_negotiation.cc:407] Connection header received
0928 02:09:12.318910 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:12.318920 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:12.319184 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:12.319352 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:12.320202 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.321747 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.324326 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.325677 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.325814 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:12.326020 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:12.326593 (+ 573us) server_negotiation.cc:299] Negotiation successful
0928 02:09:12.326775 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":312,"threads_started":1}
I20260928 02:09:12.330435 19108 tablet_service.cc:1467] Processing CreateTablet for tablet 4f635fb6b29f4ee1a285044553e7ef9e (DEFAULT_TABLE table=test-table [id=2e32ee2007504c7393ac5ab68b884828]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:09:12.331046 19243 tablet_service.cc:1467] Processing CreateTablet for tablet 4f635fb6b29f4ee1a285044553e7ef9e (DEFAULT_TABLE table=test-table [id=2e32ee2007504c7393ac5ab68b884828]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:09:12.331051 19378 tablet_service.cc:1467] Processing CreateTablet for tablet 4f635fb6b29f4ee1a285044553e7ef9e (DEFAULT_TABLE table=test-table [id=2e32ee2007504c7393ac5ab68b884828]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:09:12.331756 19108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f635fb6b29f4ee1a285044553e7ef9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:12.332444 19378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f635fb6b29f4ee1a285044553e7ef9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:12.332435 19243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f635fb6b29f4ee1a285044553e7ef9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:12.343154 19463 tablet_bootstrap.cc:492] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac: Bootstrap starting.
I20260928 02:09:12.346342 19463 tablet_bootstrap.cc:654] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:12.346753 19464 tablet_bootstrap.cc:492] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af: Bootstrap starting.
I20260928 02:09:12.347862 19463 log.cc:824] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:12.348434 19465 tablet_bootstrap.cc:492] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70: Bootstrap starting.
I20260928 02:09:12.350637 19464 tablet_bootstrap.cc:654] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:12.351706 19465 tablet_bootstrap.cc:654] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:12.351876 19464 log.cc:824] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:12.352694 19463 tablet_bootstrap.cc:492] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac: No bootstrap required, opened a new log
I20260928 02:09:12.352839 19465 log.cc:824] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:12.352999 19463 ts_tablet_manager.cc:1397] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 02:09:12.354959 19464 tablet_bootstrap.cc:492] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af: No bootstrap required, opened a new log
I20260928 02:09:12.355276 19464 ts_tablet_manager.cc:1397] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260928 02:09:12.355450 19465 tablet_bootstrap.cc:492] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70: No bootstrap required, opened a new log
I20260928 02:09:12.355695 19465 ts_tablet_manager.cc:1397] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 02:09:12.363425 19465 raft_consensus.cc:348] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d8ccb11db42f2b61c9ab2d111cf70" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37273 } } peers { permanent_uuid: "765011ee0b9b4156b75513a6144d93ac" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40417 } } peers { permanent_uuid: "d57f0e88f92b44038517c42f0cff68af" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37603 } }
I20260928 02:09:12.363806 19465 raft_consensus.cc:374] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:12.363902 19465 raft_consensus.cc:729] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105d8ccb11db42f2b61c9ab2d111cf70, State: Initialized, Role: FOLLOWER
I20260928 02:09:12.363961 19463 raft_consensus.cc:348] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d8ccb11db42f2b61c9ab2d111cf70" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37273 } } peers { permanent_uuid: "765011ee0b9b4156b75513a6144d93ac" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40417 } } peers { permanent_uuid: "d57f0e88f92b44038517c42f0cff68af" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37603 } }
I20260928 02:09:12.364395 19463 raft_consensus.cc:374] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:12.364503 19463 raft_consensus.cc:729] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 765011ee0b9b4156b75513a6144d93ac, State: Initialized, Role: FOLLOWER
I20260928 02:09:12.364619 19465 consensus_queue.cc:260] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d8ccb11db42f2b61c9ab2d111cf70" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37273 } } peers { permanent_uuid: "765011ee0b9b4156b75513a6144d93ac" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40417 } } peers { permanent_uuid: "d57f0e88f92b44038517c42f0cff68af" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37603 } }
I20260928 02:09:12.365089 19463 consensus_queue.cc:260] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d8ccb11db42f2b61c9ab2d111cf70" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37273 } } peers { permanent_uuid: "765011ee0b9b4156b75513a6144d93ac" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40417 } } peers { permanent_uuid: "d57f0e88f92b44038517c42f0cff68af" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37603 } }
I20260928 02:09:12.365926 19444 heartbeater.cc:503] Master 127.1.182.190:43965 was elected leader, sending a full tablet report...
I20260928 02:09:12.366380 19465 ts_tablet_manager.cc:1428] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260928 02:09:12.367384 19464 raft_consensus.cc:348] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d8ccb11db42f2b61c9ab2d111cf70" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37273 } } peers { permanent_uuid: "765011ee0b9b4156b75513a6144d93ac" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40417 } } peers { permanent_uuid: "d57f0e88f92b44038517c42f0cff68af" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37603 } }
I20260928 02:09:12.367810 19464 raft_consensus.cc:374] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:12.367856 19463 ts_tablet_manager.cc:1428] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260928 02:09:12.367903 19464 raft_consensus.cc:729] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57f0e88f92b44038517c42f0cff68af, State: Initialized, Role: FOLLOWER
I20260928 02:09:12.368582 19464 consensus_queue.cc:260] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d8ccb11db42f2b61c9ab2d111cf70" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37273 } } peers { permanent_uuid: "765011ee0b9b4156b75513a6144d93ac" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40417 } } peers { permanent_uuid: "d57f0e88f92b44038517c42f0cff68af" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37603 } }
I20260928 02:09:12.369879 19309 heartbeater.cc:503] Master 127.1.182.190:43965 was elected leader, sending a full tablet report...
I20260928 02:09:12.370491 19464 ts_tablet_manager.cc:1428] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260928 02:09:12.418368 19174 tablet.cc:2369] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:09:12.460857 19310 tablet.cc:2369] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:09:12.487323 19445 tablet.cc:2369] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:09:12.572387 19469 raft_consensus.cc:482] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:09:12.572676 19469 raft_consensus.cc:504] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d8ccb11db42f2b61c9ab2d111cf70" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37273 } } peers { permanent_uuid: "765011ee0b9b4156b75513a6144d93ac" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40417 } } peers { permanent_uuid: "d57f0e88f92b44038517c42f0cff68af" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37603 } }
I20260928 02:09:12.574415 19469 leader_election.cc:288] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 765011ee0b9b4156b75513a6144d93ac (127.1.182.129:40417), d57f0e88f92b44038517c42f0cff68af (127.1.182.130:37603)
I20260928 02:09:12.580874 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:12.575229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:38015 (local address 127.1.182.130:37603)
0928 02:09:12.575449 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:12.575456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:12.575475 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:09:12.575553 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:12.575558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:12.575634 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:12.575732 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:12.576237 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.577159 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.578982 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.579964 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.580091 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:12.580142 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:12.580495 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 02:09:12.580582 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:09:12.580888 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:12.574706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:37603 (local address 127.1.182.131:38015)
0928 02:09:12.574945 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:12.574966 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:12.575146 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:12.575806 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:12.575814 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:12.575864 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:12.576093 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:12.576109 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.577285 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.577291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:12.578820 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:12.578832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.580113 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.580119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:12.580322 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:12.580359 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:09:12.580454 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 02:09:12.580584 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":42}
I20260928 02:09:12.581560 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:12.574881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:39031 (local address 127.1.182.129:40417)
0928 02:09:12.575159 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:12.575165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:12.575781 (+ 616us) server_negotiation.cc:407] Connection header received
0928 02:09:12.575947 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:12.575952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:12.576031 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:12.576151 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:12.576772 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.577655 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.579553 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.580549 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.580691 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:12.580771 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:12.581185 (+ 414us) server_negotiation.cc:299] Negotiation successful
0928 02:09:12.581313 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 02:09:12.581669 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:12.574799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:40417 (local address 127.1.182.131:39031)
0928 02:09:12.575676 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:12.575695 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:12.575804 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:12.576179 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:12.576185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:12.576212 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:12.576605 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:12.576617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.577803 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.577807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:12.579400 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:12.579416 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.580759 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.580766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:12.580952 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:12.580987 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:09:12.581158 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 02:09:12.581285 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":1,"thread_start_us":536,"threads_started":1}
I20260928 02:09:12.582100 19264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f635fb6b29f4ee1a285044553e7ef9e" candidate_uuid: "105d8ccb11db42f2b61c9ab2d111cf70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d57f0e88f92b44038517c42f0cff68af" is_pre_election: true
I20260928 02:09:12.582679 19264 raft_consensus.cc:2454] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 105d8ccb11db42f2b61c9ab2d111cf70 in term 0.
I20260928 02:09:12.582716 19128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f635fb6b29f4ee1a285044553e7ef9e" candidate_uuid: "105d8ccb11db42f2b61c9ab2d111cf70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "765011ee0b9b4156b75513a6144d93ac" is_pre_election: true
I20260928 02:09:12.583258 19128 raft_consensus.cc:2454] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 105d8ccb11db42f2b61c9ab2d111cf70 in term 0.
I20260928 02:09:12.583348 19332 leader_election.cc:302] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 105d8ccb11db42f2b61c9ab2d111cf70, d57f0e88f92b44038517c42f0cff68af; no voters:
I20260928 02:09:12.583817 19469 raft_consensus.cc:2781] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:09:12.583962 19469 raft_consensus.cc:482] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:09:12.584075 19469 raft_consensus.cc:3037] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:12.586234 19469 raft_consensus.cc:504] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d8ccb11db42f2b61c9ab2d111cf70" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37273 } } peers { permanent_uuid: "765011ee0b9b4156b75513a6144d93ac" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40417 } } peers { permanent_uuid: "d57f0e88f92b44038517c42f0cff68af" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37603 } }
I20260928 02:09:12.586840 19469 leader_election.cc:288] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 [CANDIDATE]: Term 1 election: Requested vote from peers 765011ee0b9b4156b75513a6144d93ac (127.1.182.129:40417), d57f0e88f92b44038517c42f0cff68af (127.1.182.130:37603)
I20260928 02:09:12.587214 19128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f635fb6b29f4ee1a285044553e7ef9e" candidate_uuid: "105d8ccb11db42f2b61c9ab2d111cf70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "765011ee0b9b4156b75513a6144d93ac"
I20260928 02:09:12.587419 19128 raft_consensus.cc:3037] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:12.587406 19264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f635fb6b29f4ee1a285044553e7ef9e" candidate_uuid: "105d8ccb11db42f2b61c9ab2d111cf70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d57f0e88f92b44038517c42f0cff68af"
I20260928 02:09:12.587639 19264 raft_consensus.cc:3037] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:12.589833 19128 raft_consensus.cc:2454] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 105d8ccb11db42f2b61c9ab2d111cf70 in term 1.
I20260928 02:09:12.590054 19264 raft_consensus.cc:2454] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 105d8ccb11db42f2b61c9ab2d111cf70 in term 1.
I20260928 02:09:12.590270 19331 leader_election.cc:302] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 105d8ccb11db42f2b61c9ab2d111cf70, 765011ee0b9b4156b75513a6144d93ac; no voters:
I20260928 02:09:12.590554 19469 raft_consensus.cc:2781] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:12.591039 19469 raft_consensus.cc:686] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 [term 1 LEADER]: Becoming Leader. State: Replica: 105d8ccb11db42f2b61c9ab2d111cf70, State: Running, Role: LEADER
I20260928 02:09:12.591596 19469 consensus_queue.cc:237] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d8ccb11db42f2b61c9ab2d111cf70" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37273 } } peers { permanent_uuid: "765011ee0b9b4156b75513a6144d93ac" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40417 } } peers { permanent_uuid: "d57f0e88f92b44038517c42f0cff68af" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37603 } }
I20260928 02:09:12.596416 18980 catalog_manager.cc:5495] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 105d8ccb11db42f2b61c9ab2d111cf70 (127.1.182.131). New cstate: current_term: 1 leader_uuid: "105d8ccb11db42f2b61c9ab2d111cf70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d8ccb11db42f2b61c9ab2d111cf70" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "765011ee0b9b4156b75513a6144d93ac" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d57f0e88f92b44038517c42f0cff68af" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37603 } health_report { overall_health: UNKNOWN } } }
I20260928 02:09:12.617422 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:12.611898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48910 (local address 127.1.182.131:37273)
0928 02:09:12.612129 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:12.612135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:12.612155 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:09:12.612318 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:12.612322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:12.612414 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:12.612600 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:09:12.612979 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.613711 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.615234 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.615647 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.615789 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:12.616318 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:09:12.616834 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:09:12.617002 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:12.617067 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 02:09:12.617149 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:09:12.626209 19264 raft_consensus.cc:1261] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af [term 1 FOLLOWER]: Refusing update from remote peer 105d8ccb11db42f2b61c9ab2d111cf70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:09:12.626255 19128 raft_consensus.cc:1261] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac [term 1 FOLLOWER]: Refusing update from remote peer 105d8ccb11db42f2b61c9ab2d111cf70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:09:12.627239 19473 consensus_queue.cc:1035] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "765011ee0b9b4156b75513a6144d93ac" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 40417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:09:12.627619 19469 consensus_queue.cc:1035] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d57f0e88f92b44038517c42f0cff68af" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:09:12.640230 19476 mvcc.cc:205] Tried to move back new op lower bound from 7334139300346445824 to 7334139300218597376. Current Snapshot: MvccSnapshot[applied={T|T < 7334139300346445824}]
I20260928 02:09:12.641482 19477 mvcc.cc:205] Tried to move back new op lower bound from 7334139300346445824 to 7334139300218597376. Current Snapshot: MvccSnapshot[applied={T|T < 7334139300346445824}]
I20260928 02:09:12.654187 18982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:09:12.654502 18982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:09:12.658319 18982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 121ee6244c9345a792acb98aa8de6cd5: Sending DeleteTablet for 3 replicas of tablet 4f635fb6b29f4ee1a285044553e7ef9e
I20260928 02:09:12.659523 19108 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f635fb6b29f4ee1a285044553e7ef9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:09:12 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:33748
I20260928 02:09:12.659621 19378 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f635fb6b29f4ee1a285044553e7ef9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:09:12 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:48906
I20260928 02:09:12.660584 19243 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f635fb6b29f4ee1a285044553e7ef9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:09:12 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:46654
I20260928 02:09:12.661152 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 19044
I20260928 02:09:12.674773 19486 tablet_replica.cc:330] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac: stopping tablet replica
I20260928 02:09:12.675310 19486 raft_consensus.cc:2229] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:09:12.675894 19486 raft_consensus.cc:2258] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:09:12.676862 19488 tablet_replica.cc:330] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af: stopping tablet replica
I20260928 02:09:12.677326 19488 raft_consensus.cc:2229] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:09:12.677846 19488 raft_consensus.cc:2258] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:09:12.679571 19486 ts_tablet_manager.cc:1905] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:09:12.682869 19487 tablet_replica.cc:330] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70: stopping tablet replica
I20260928 02:09:12.683359 19487 raft_consensus.cc:2229] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:09:12.684160 19487 raft_consensus.cc:2258] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:09:12.685591 19488 ts_tablet_manager.cc:1905] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 02:09:12 dist-test-slave-23bp krb5kdc[18937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:09:12.687518 19486 ts_tablet_manager.cc:1918] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:09:12.687678 19486 log.cc:1196] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-0/wal/wals/4f635fb6b29f4ee1a285044553e7ef9e
I20260928 02:09:12.688305 19486 ts_tablet_manager.cc:1939] T 4f635fb6b29f4ee1a285044553e7ef9e P 765011ee0b9b4156b75513a6144d93ac: Deleting consensus metadata
I20260928 02:09:12.690516 18966 catalog_manager.cc:4841] TS 765011ee0b9b4156b75513a6144d93ac (127.1.182.129:40417): tablet 4f635fb6b29f4ee1a285044553e7ef9e (table test-table [id=2e32ee2007504c7393ac5ab68b884828]) successfully deleted
I20260928 02:09:12.692356 19487 ts_tablet_manager.cc:1905] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:09:12.692634 19488 ts_tablet_manager.cc:1918] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:09:12.692778 19488 log.cc:1196] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/4f635fb6b29f4ee1a285044553e7ef9e
I20260928 02:09:12.693328 19488 ts_tablet_manager.cc:1939] T 4f635fb6b29f4ee1a285044553e7ef9e P d57f0e88f92b44038517c42f0cff68af: Deleting consensus metadata
I20260928 02:09:12.695176 18967 catalog_manager.cc:4841] TS d57f0e88f92b44038517c42f0cff68af (127.1.182.130:37603): tablet 4f635fb6b29f4ee1a285044553e7ef9e (table test-table [id=2e32ee2007504c7393ac5ab68b884828]) successfully deleted
I20260928 02:09:12.697520 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:12.662929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33754 (local address 127.1.182.129:40417)
0928 02:09:12.663202 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:12.663212 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:12.673839 (+ 10627us) server_negotiation.cc:407] Connection header received
0928 02:09:12.674451 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:12.674463 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:12.674582 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:12.674662 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:12.677195 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.678266 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.680829 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.681498 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.681601 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:12.689106 (+ 7505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:12.689126 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:12.689147 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:12.689212 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:12.693398 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:12.694548 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:12.694558 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:12.694562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:12.694721 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:12.695264 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:12.695269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:12.695273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:12.695856 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:12.696167 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:12.696808 (+ 641us) server_negotiation.cc:299] Negotiation successful
0928 02:09:12.697131 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 02:09:12.698490 19168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:12.699520 19487 ts_tablet_manager.cc:1918] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:09:12.699713 19487 log.cc:1196] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/4f635fb6b29f4ee1a285044553e7ef9e
I20260928 02:09:12.700344 19487 ts_tablet_manager.cc:1939] T 4f635fb6b29f4ee1a285044553e7ef9e P 105d8ccb11db42f2b61c9ab2d111cf70: Deleting consensus metadata
I20260928 02:09:12.702816 18967 catalog_manager.cc:4841] TS 105d8ccb11db42f2b61c9ab2d111cf70 (127.1.182.131:37273): tablet 4f635fb6b29f4ee1a285044553e7ef9e (table test-table [id=2e32ee2007504c7393ac5ab68b884828]) successfully deleted
I20260928 02:09:12.789553 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 19044
I20260928 02:09:12.807585 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 19179
Sep 28 02:09:12 dist-test-slave-23bp krb5kdc[18937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.130@KRBTEST.COM
I20260928 02:09:12.825243 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:12.808450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46666 (local address 127.1.182.130:37603)
0928 02:09:12.808638 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:12.808642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:12.808660 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:09:12.808859 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:12.808864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:12.808926 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:12.809012 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:12.810645 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.811536 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.813574 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.813939 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.814041 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:12.819394 (+ 5353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:12.819409 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:12.819426 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:12.819460 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:12.822331 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:12.823025 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:12.823032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:12.823034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:12.823282 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:12.823657 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:12.823665 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:12.823671 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:12.824060 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:12.824384 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:12.824667 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 02:09:12.824886 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 02:09:12.825897 19304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:12.918128 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 19179
I20260928 02:09:12.935602 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 19315
Sep 28 02:09:12 dist-test-slave-23bp krb5kdc[18937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.131@KRBTEST.COM
I20260928 02:09:12.954561 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:12.936699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48918 (local address 127.1.182.131:37273)
0928 02:09:12.936927 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:12.936933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:12.936955 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:09:12.937026 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:12.937032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:12.937141 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:12.937263 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:12.938498 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.939823 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.941410 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:12.942099 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:12.942231 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:12.946702 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:12.946724 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:12.946745 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:12.946800 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:12.950945 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:12.951995 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:12.952006 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:12.952010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:12.952231 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:12.952691 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:12.952698 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:12.952706 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:12.953221 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:12.953552 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:12.953855 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 02:09:12.954107 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:09:12.955538 19439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:13.083616 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 19315
I20260928 02:09:13.103055 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 18950
I20260928 02:09:13.104271 19013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:13.216552 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 18950
2026-09-28T02:09:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4555 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T02:09:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:09:13Z Disabled control of system clock
I20260928 02:09:13.299230 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:42147
--webserver_interface=127.1.182.190
--webserver_port=0
--builtin_ntp_servers=127.1.182.148:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:42147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:09:13.635677 19498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:13.635986 19498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:13.636153 19498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:13.643579 19498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:09:13.643690 19498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:13.643779 19498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:13.643891 19498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:09:13.643971 19498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:09:13.653545 19498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:42147
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:42147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:13.654964 19498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:13.656822 19498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:13.668736 19503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:13.668670 19506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:13.668709 19504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:13.670151 19498 server_base.cc:1034] running on GCE node
I20260928 02:09:13.670979 19498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:13.672564 19498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:13.673787 19498 hybrid_clock.cc:648] HybridClock initialized: now 1790561353673722 us; error 60 us; skew 500 ppm
I20260928 02:09:13.674135 19498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:09:13.680431 19498 webserver.cc:460] Webserver started at http://127.1.182.190:45167/ using document root <none> and password file <none>
I20260928 02:09:13.681185 19498 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:13.681319 19498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:13.681641 19498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:13.684756 19498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "cb8bd5c6035a4727bbdf2c4692b4f1de"
format_stamp: "Formatted at 2026-09-28 02:09:13 on dist-test-slave-23bp"
server_key: "af11a407524c812a951d5f5716df140b"
server_key_iv: "7dcc8f497c90ccf9a44c6aa03b675bd2"
server_key_version: "encryptionkey@0"
I20260928 02:09:13.685571 19498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "cb8bd5c6035a4727bbdf2c4692b4f1de"
format_stamp: "Formatted at 2026-09-28 02:09:13 on dist-test-slave-23bp"
server_key: "af11a407524c812a951d5f5716df140b"
server_key_iv: "7dcc8f497c90ccf9a44c6aa03b675bd2"
server_key_version: "encryptionkey@0"
I20260928 02:09:13.690877 19498 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:09:13.694406 19512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:13.695875 19498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:09:13.696070 19498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "cb8bd5c6035a4727bbdf2c4692b4f1de"
format_stamp: "Formatted at 2026-09-28 02:09:13 on dist-test-slave-23bp"
server_key: "af11a407524c812a951d5f5716df140b"
server_key_iv: "7dcc8f497c90ccf9a44c6aa03b675bd2"
server_key_version: "encryptionkey@0"
I20260928 02:09:13.696285 19498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:13.711220 19498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:13.712037 19498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:13.712330 19498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:13.730909 19498 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:42147
I20260928 02:09:13.730929 19563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:42147 every 8 connection(s)
I20260928 02:09:13.732311 19498 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:09:13.733358 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 19498
I20260928 02:09:13.733565 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:09:13.734112 1754 external_mini_cluster.cc:1428] Setting key 853b8e2d7866ab00bf37757d3cf53e21
I20260928 02:09:13.736914 19564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:13.747999 19567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:13.736080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33446 (local address 127.1.182.190:42147)
0928 02:09:13.737138 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:13.737148 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:13.737192 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:09:13.738797 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:13.738818 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:13.739353 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:13.739689 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:13.740271 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:13.741586 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:13.743544 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:13.744210 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:13.744336 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:13.745630 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:13.745640 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:13.745653 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:13.746211 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:13.746288 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:13.746311 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:13.746402 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:13.746660 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 02:09:13.747011 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":381,"threads_started":1}
I20260928 02:09:13.749161 19564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de: Bootstrap starting.
I20260928 02:09:13.752246 19564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:13.753214 19564 log.cc:824] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:13.755923 19564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de: No bootstrap required, opened a new log
I20260928 02:09:13.763131 19564 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8bd5c6035a4727bbdf2c4692b4f1de" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 42147 } }
I20260928 02:09:13.763520 19564 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:13.763635 19564 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8bd5c6035a4727bbdf2c4692b4f1de, State: Initialized, Role: FOLLOWER
I20260928 02:09:13.764200 19564 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8bd5c6035a4727bbdf2c4692b4f1de" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 42147 } }
I20260928 02:09:13.764434 19564 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:09:13.764568 19564 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:09:13.764743 19564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:13.766839 19564 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8bd5c6035a4727bbdf2c4692b4f1de" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 42147 } }
I20260928 02:09:13.767391 19564 leader_election.cc:302] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb8bd5c6035a4727bbdf2c4692b4f1de; no voters:
I20260928 02:09:13.768064 19564 leader_election.cc:288] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:09:13.768246 19569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:13.769223 19569 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de [term 1 LEADER]: Becoming Leader. State: Replica: cb8bd5c6035a4727bbdf2c4692b4f1de, State: Running, Role: LEADER
I20260928 02:09:13.769778 19569 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8bd5c6035a4727bbdf2c4692b4f1de" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 42147 } }
I20260928 02:09:13.770773 19564 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:09:13.772994 19571 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb8bd5c6035a4727bbdf2c4692b4f1de. Latest consensus state: current_term: 1 leader_uuid: "cb8bd5c6035a4727bbdf2c4692b4f1de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8bd5c6035a4727bbdf2c4692b4f1de" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 42147 } } }
I20260928 02:09:13.773295 19570 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb8bd5c6035a4727bbdf2c4692b4f1de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8bd5c6035a4727bbdf2c4692b4f1de" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 42147 } } }
I20260928 02:09:13.773481 19571 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:13.773594 19570 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:13.775198 19579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:09:13.780694 19579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:09:13.790046 19579 catalog_manager.cc:1347] Generated new cluster ID: fed6a959d19b409197d35c849f2bd892
I20260928 02:09:13.790190 19579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:09:13.810395 19579 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:09:13.811789 19579 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:09:13.823570 19579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de: Generated new TSK 0
I20260928 02:09:13.824671 19579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:09:13.838241 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:42147
--builtin_ntp_servers=127.1.182.148:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:09:14.201633 19589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:14.201936 19589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:14.202040 19589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:14.209865 19589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:14.209973 19589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:14.210079 19589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:09:14.219246 19589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:42147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:14.220481 19589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:14.222258 19589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:14.235559 19597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:14.235625 19595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:14.235646 19594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:14.240753 19589 server_base.cc:1034] running on GCE node
I20260928 02:09:14.241367 19589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:14.242307 19589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:14.243513 19589 hybrid_clock.cc:648] HybridClock initialized: now 1790561354243466 us; error 33 us; skew 500 ppm
I20260928 02:09:14.243803 19589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:09:14.246035 19589 webserver.cc:460] Webserver started at http://127.1.182.129:40473/ using document root <none> and password file <none>
I20260928 02:09:14.246739 19589 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:14.246825 19589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:14.247241 19589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:14.250883 19589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "f074ef8003bf4826885ed16d0b834c2c"
format_stamp: "Formatted at 2026-09-28 02:09:14 on dist-test-slave-23bp"
server_key: "4a11e8d64a13526fee262a249b536a5f"
server_key_iv: "640c494c6e81361f3c6888e1d5b8540f"
server_key_version: "encryptionkey@0"
I20260928 02:09:14.251698 19589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "f074ef8003bf4826885ed16d0b834c2c"
format_stamp: "Formatted at 2026-09-28 02:09:14 on dist-test-slave-23bp"
server_key: "4a11e8d64a13526fee262a249b536a5f"
server_key_iv: "640c494c6e81361f3c6888e1d5b8540f"
server_key_version: "encryptionkey@0"
I20260928 02:09:14.256876 19589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:09:14.260370 19603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:14.261739 19589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:09:14.261914 19589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "f074ef8003bf4826885ed16d0b834c2c"
format_stamp: "Formatted at 2026-09-28 02:09:14 on dist-test-slave-23bp"
server_key: "4a11e8d64a13526fee262a249b536a5f"
server_key_iv: "640c494c6e81361f3c6888e1d5b8540f"
server_key_version: "encryptionkey@0"
I20260928 02:09:14.262131 19589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:14.293867 19589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:14.294806 19589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:14.295209 19589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:14.296243 19589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:09:14.297724 19589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:09:14.297856 19589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:14.297997 19589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:09:14.298105 19589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:14.331275 19589 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:41053
I20260928 02:09:14.331367 19716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:41053 every 8 connection(s)
I20260928 02:09:14.332607 19589 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:09:14.333297 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 19589
I20260928 02:09:14.333491 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:09:14.334107 1754 external_mini_cluster.cc:1428] Setting key 603bc2fc60397845c40c000eb1794075
I20260928 02:09:14.337368 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:42147
--builtin_ntp_servers=127.1.182.148:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:09:14.345860 19719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:14.335484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:43727 (local address 127.1.182.190:42147)
0928 02:09:14.335934 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:14.335940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:14.337354 (+ 1414us) server_negotiation.cc:407] Connection header received
0928 02:09:14.339285 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:14.339291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:14.339377 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:14.339510 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:14.340600 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:14.341310 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:14.342915 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:14.343415 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:14.343558 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:14.344976 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:14.344982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:14.344988 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:14.345019 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:14.345064 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:14.345076 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:14.345168 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:14.345397 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 02:09:14.345499 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":215,"threads_started":1}
I20260928 02:09:14.346859 19720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:14.336230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:42147 (local address 127.1.182.129:43727)
0928 02:09:14.337205 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:14.337243 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:14.338956 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:14.339759 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:14.339768 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:14.339860 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:09:14.340428 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:14.340444 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:14.341470 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:14.341476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:14.342758 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:14.342770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:14.343650 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:14.343657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:14.343904 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:14.344656 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:09:14.344658 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:09:14.344698 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:09:14.344775 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:09:14.345183 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:09:14.345195 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:09:14.345203 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:09:14.345386 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 02:09:14.345794 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":316,"threads_started":1}
I20260928 02:09:14.348474 19717 heartbeater.cc:347] Connected to a master server at 127.1.182.190:42147
I20260928 02:09:14.348796 19717 heartbeater.cc:464] Registering TS with master...
I20260928 02:09:14.349716 19717 heartbeater.cc:511] Master 127.1.182.190:42147 requested a full tablet report, sending...
I20260928 02:09:14.352129 19529 ts_manager.cc:194] Registered new tserver with Master: f074ef8003bf4826885ed16d0b834c2c (127.1.182.129:41053)
I20260928 02:09:14.354233 19529 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.182.129:43727
W20260928 02:09:14.673920 19721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:14.674201 19721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:14.674288 19721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:14.681556 19721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:14.681658 19721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:14.681826 19721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:09:14.690840 19721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:42147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:14.692175 19721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:14.693874 19721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:14.706966 19726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:14.706928 19727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:14.707012 19729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:14.712774 19721 server_base.cc:1034] running on GCE node
I20260928 02:09:14.713469 19721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:14.714452 19721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:14.715687 19721 hybrid_clock.cc:648] HybridClock initialized: now 1790561354715638 us; error 59 us; skew 500 ppm
I20260928 02:09:14.716022 19721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:09:14.718382 19721 webserver.cc:460] Webserver started at http://127.1.182.130:41877/ using document root <none> and password file <none>
I20260928 02:09:14.719175 19721 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:14.719307 19721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:14.719635 19721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:14.722520 19721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "8969eda00dc44d09964a0bad9ca4e813"
format_stamp: "Formatted at 2026-09-28 02:09:14 on dist-test-slave-23bp"
server_key: "6ae4f5194adaf9112dec68ae2df82582"
server_key_iv: "d5fe92e78e22aaf06f5d17faf9470b5d"
server_key_version: "encryptionkey@0"
I20260928 02:09:14.723376 19721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "8969eda00dc44d09964a0bad9ca4e813"
format_stamp: "Formatted at 2026-09-28 02:09:14 on dist-test-slave-23bp"
server_key: "6ae4f5194adaf9112dec68ae2df82582"
server_key_iv: "d5fe92e78e22aaf06f5d17faf9470b5d"
server_key_version: "encryptionkey@0"
I20260928 02:09:14.728425 19721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 02:09:14.732091 19735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:14.733558 19721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:09:14.733721 19721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "8969eda00dc44d09964a0bad9ca4e813"
format_stamp: "Formatted at 2026-09-28 02:09:14 on dist-test-slave-23bp"
server_key: "6ae4f5194adaf9112dec68ae2df82582"
server_key_iv: "d5fe92e78e22aaf06f5d17faf9470b5d"
server_key_version: "encryptionkey@0"
I20260928 02:09:14.733960 19721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:14.752758 19721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:14.753661 19721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:14.753961 19721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:14.754951 19721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:09:14.756425 19721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:09:14.756512 19721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:14.756669 19721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:09:14.756760 19721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:14.790859 19721 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:42987
I20260928 02:09:14.790889 19847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:42987 every 8 connection(s)
I20260928 02:09:14.792279 19721 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:09:14.794503 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 19721
I20260928 02:09:14.794694 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:09:14.795342 1754 external_mini_cluster.cc:1428] Setting key 40cedf3360f0d33b07c6428407d20fa8
I20260928 02:09:14.798436 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:42147
--builtin_ntp_servers=127.1.182.148:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:09:14.806032 19719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:14.794987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:37777 (local address 127.1.182.190:42147)
0928 02:09:14.795250 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:14.795258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:14.796997 (+ 1739us) server_negotiation.cc:407] Connection header received
0928 02:09:14.799013 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:14.799019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:14.799136 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:14.799267 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:14.800438 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:14.801426 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:14.803104 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:14.803536 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:14.803679 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:14.805189 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:14.805194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:14.805197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:14.805216 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:14.805247 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:14.805259 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:14.805357 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:14.805607 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 02:09:14.805709 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260928 02:09:14.807258 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:14.795663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:42147 (local address 127.1.182.130:37777)
0928 02:09:14.796846 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:14.796883 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:14.798728 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:14.799525 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:14.799534 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:14.799632 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:09:14.800253 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:14.800270 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:14.801595 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:14.801605 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:14.802921 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:14.802934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:14.803772 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:14.803779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:14.804034 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:14.804791 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:09:14.804794 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:09:14.804836 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:09:14.804922 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:09:14.805356 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:09:14.805368 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:09:14.805378 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:09:14.805597 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 02:09:14.806035 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":400,"threads_started":1}
I20260928 02:09:14.808823 19848 heartbeater.cc:347] Connected to a master server at 127.1.182.190:42147
I20260928 02:09:14.809183 19848 heartbeater.cc:464] Registering TS with master...
I20260928 02:09:14.809952 19848 heartbeater.cc:511] Master 127.1.182.190:42147 requested a full tablet report, sending...
I20260928 02:09:14.812443 19529 ts_manager.cc:194] Registered new tserver with Master: 8969eda00dc44d09964a0bad9ca4e813 (127.1.182.130:42987)
I20260928 02:09:14.813666 19529 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.182.130:37777
W20260928 02:09:15.128852 19851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:15.129140 19851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:15.129226 19851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:15.136526 19851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:15.136632 19851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:15.136739 19851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:09:15.146003 19851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:42147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:15.147279 19851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:15.149024 19851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:15.162197 19857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:15.162269 19856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:15.162241 19859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:15.163022 19851 server_base.cc:1034] running on GCE node
I20260928 02:09:15.163743 19851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:15.164647 19851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:15.165844 19851 hybrid_clock.cc:648] HybridClock initialized: now 1790561355165805 us; error 28 us; skew 500 ppm
I20260928 02:09:15.166183 19851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:09:15.168470 19851 webserver.cc:460] Webserver started at http://127.1.182.131:37933/ using document root <none> and password file <none>
I20260928 02:09:15.169168 19851 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:15.169286 19851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:15.169615 19851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:15.172595 19851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "7367e54fdb0244ba9481a8903980d336"
format_stamp: "Formatted at 2026-09-28 02:09:15 on dist-test-slave-23bp"
server_key: "829fe1db2d0d3693e5cf8dcaa9ea50c0"
server_key_iv: "4afc71c4f72d75c62c0359fdacf92901"
server_key_version: "encryptionkey@0"
I20260928 02:09:15.173365 19851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "7367e54fdb0244ba9481a8903980d336"
format_stamp: "Formatted at 2026-09-28 02:09:15 on dist-test-slave-23bp"
server_key: "829fe1db2d0d3693e5cf8dcaa9ea50c0"
server_key_iv: "4afc71c4f72d75c62c0359fdacf92901"
server_key_version: "encryptionkey@0"
I20260928 02:09:15.178421 19851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 02:09:15.181766 19866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:15.183029 19851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:09:15.183247 19851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "7367e54fdb0244ba9481a8903980d336"
format_stamp: "Formatted at 2026-09-28 02:09:15 on dist-test-slave-23bp"
server_key: "829fe1db2d0d3693e5cf8dcaa9ea50c0"
server_key_iv: "4afc71c4f72d75c62c0359fdacf92901"
server_key_version: "encryptionkey@0"
I20260928 02:09:15.183446 19851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:15.205500 19851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:15.206332 19851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:15.206638 19851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:15.207736 19851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:09:15.209177 19851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:09:15.209286 19851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:15.209411 19851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:09:15.209503 19851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:15.242151 19851 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:40495
I20260928 02:09:15.242237 19978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:40495 every 8 connection(s)
I20260928 02:09:15.243556 19851 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:09:15.244913 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 19851
I20260928 02:09:15.245107 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:09:15.245607 1754 external_mini_cluster.cc:1428] Setting key a8b5cbf107271cb9cfe5a7e083c07aea
I20260928 02:09:15.255877 19719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.246307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:57167 (local address 127.1.182.190:42147)
0928 02:09:15.246494 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:15.246499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:15.248369 (+ 1870us) server_negotiation.cc:407] Connection header received
0928 02:09:15.250154 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:15.250158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:15.250221 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:15.250299 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:15.251321 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.252041 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.253492 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.253861 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.253949 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:15.255184 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:15.255189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:15.255192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:15.255209 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:15.255239 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:15.255251 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:15.255322 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:15.255511 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 02:09:15.255584 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:09:15.256752 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.247020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:42147 (local address 127.1.182.131:57167)
0928 02:09:15.248205 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:15.248256 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:15.249926 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:15.250457 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:15.250465 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:15.250556 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:09:15.251183 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:15.251201 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.252182 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.252186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:15.253376 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:15.253386 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.254034 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.254038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:15.254230 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:15.254829 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:09:15.254831 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:09:15.254867 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:09:15.254953 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:09:15.255334 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:09:15.255342 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:09:15.255347 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:09:15.255484 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 02:09:15.255759 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":403,"threads_started":1}
I20260928 02:09:15.258010 19979 heartbeater.cc:347] Connected to a master server at 127.1.182.190:42147
I20260928 02:09:15.258329 19979 heartbeater.cc:464] Registering TS with master...
I20260928 02:09:15.259177 19979 heartbeater.cc:511] Master 127.1.182.190:42147 requested a full tablet report, sending...
I20260928 02:09:15.260679 19528 ts_manager.cc:194] Registered new tserver with Master: 7367e54fdb0244ba9481a8903980d336 (127.1.182.131:40495)
I20260928 02:09:15.261575 19528 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.182.131:57167
I20260928 02:09:15.273133 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:09:15.283474 19719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.277285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33454 (local address 127.1.182.190:42147)
0928 02:09:15.277503 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:15.277512 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:15.277895 (+ 383us) server_negotiation.cc:407] Connection header received
0928 02:09:15.278120 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:15.278131 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:15.278207 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:15.278316 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:15.278844 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.279617 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.281001 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.281426 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.281562 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:15.282717 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:15.282723 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:15.282726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:15.282744 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:15.282778 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:15.282790 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:15.282887 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:15.283034 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 02:09:15.283164 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:09:15.287626 19528 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:09:15.294431 19528 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:09:15.328388 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.306889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:41053 (local address 127.0.0.1:39588)
0928 02:09:15.318821 (+ 11932us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:15.318839 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:15.318959 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:15.319916 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:15.319920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:15.319946 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:15.320417 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:15.320429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.322382 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.322388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:15.325475 (+ 3087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:15.325492 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.327351 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.327357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:15.327571 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:15.327628 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 02:09:15.327779 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 02:09:15.327969 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11815,"mutex_wait_us":234,"thread_start_us":10508,"threads_started":1}
I20260928 02:09:15.328860 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.306748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:40495 (local address 127.0.0.1:48176)
0928 02:09:15.317844 (+ 11096us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:15.317885 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:15.318091 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:15.319524 (+ 1433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:15.319546 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:15.319577 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:15.320037 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:15.320053 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.322382 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.322388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:15.325475 (+ 3087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:15.325492 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.328091 (+ 2599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.328098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:15.328282 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:15.328318 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:09:15.328391 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:09:15.328523 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10796,"mutex_wait_us":209,"thread_start_us":12474,"threads_started":1}
I20260928 02:09:15.329181 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.307115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39588 (local address 127.1.182.129:41053)
0928 02:09:15.317322 (+ 10207us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:15.317336 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:15.319123 (+ 1787us) server_negotiation.cc:407] Connection header received
0928 02:09:15.319244 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:15.319252 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:15.319485 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:15.319686 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:15.320605 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.322215 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.325739 (+ 3524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.327106 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.327245 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:15.327515 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:15.328664 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0928 02:09:15.328855 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10037,"thread_start_us":9905,"threads_started":1}
I20260928 02:09:15.332629 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.318571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:42987 (local address 127.0.0.1:48664)
0928 02:09:15.325874 (+ 7303us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:15.325893 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:15.326025 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:15.326697 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:15.326701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:15.326727 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:15.327110 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:15.327123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.328993 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.328998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:15.330464 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:15.330477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.331902 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.331908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:15.332075 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:15.332105 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 02:09:15.332192 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:09:15.332313 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7179,"thread_start_us":5467,"threads_started":1}
I20260928 02:09:15.333461 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.318590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48664 (local address 127.1.182.130:42987)
0928 02:09:15.325933 (+ 7343us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:15.325952 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:15.325977 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:09:15.326209 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:15.326217 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:15.326468 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:15.326690 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:15.327292 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.328808 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.330663 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.331707 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.331865 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:15.332107 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:15.332929 (+ 822us) server_negotiation.cc:299] Negotiation successful
0928 02:09:15.333111 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7197,"thread_start_us":7081,"threads_started":1}
I20260928 02:09:15.333765 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.306833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48176 (local address 127.1.182.131:40495)
0928 02:09:15.318590 (+ 11757us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:15.318599 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:15.318640 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:09:15.318752 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:15.318760 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:15.319055 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:15.320290 (+ 1235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:15.320605 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.322214 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.326124 (+ 3910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.327856 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.329474 (+ 1618us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:15.329750 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:15.329814 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 02:09:15.329965 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11549,"thread_start_us":11401,"threads_started":1}
I20260928 02:09:15.333518 19651 tablet_service.cc:1467] Processing CreateTablet for tablet 89534a35795349e28a7cc88c2a1c63a9 (DEFAULT_TABLE table=test-table [id=24ad19c19b584e2dbf06516a977c7bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:09:15.336076 19651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89534a35795349e28a7cc88c2a1c63a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:15.337080 19914 tablet_service.cc:1467] Processing CreateTablet for tablet 89534a35795349e28a7cc88c2a1c63a9 (DEFAULT_TABLE table=test-table [id=24ad19c19b584e2dbf06516a977c7bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:09:15.337138 19783 tablet_service.cc:1467] Processing CreateTablet for tablet 89534a35795349e28a7cc88c2a1c63a9 (DEFAULT_TABLE table=test-table [id=24ad19c19b584e2dbf06516a977c7bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:09:15.338368 19914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89534a35795349e28a7cc88c2a1c63a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:15.338737 19783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89534a35795349e28a7cc88c2a1c63a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:15.350070 19997 tablet_bootstrap.cc:492] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336: Bootstrap starting.
I20260928 02:09:15.352164 19998 tablet_bootstrap.cc:492] T 89534a35795349e28a7cc88c2a1c63a9 P f074ef8003bf4826885ed16d0b834c2c: Bootstrap starting.
I20260928 02:09:15.353497 19999 tablet_bootstrap.cc:492] T 89534a35795349e28a7cc88c2a1c63a9 P 8969eda00dc44d09964a0bad9ca4e813: Bootstrap starting.
I20260928 02:09:15.354190 19997 tablet_bootstrap.cc:654] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:15.355684 19998 tablet_bootstrap.cc:654] T 89534a35795349e28a7cc88c2a1c63a9 P f074ef8003bf4826885ed16d0b834c2c: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:15.356539 19997 log.cc:824] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:15.356812 19999 tablet_bootstrap.cc:654] T 89534a35795349e28a7cc88c2a1c63a9 P 8969eda00dc44d09964a0bad9ca4e813: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:15.356949 19998 log.cc:824] T 89534a35795349e28a7cc88c2a1c63a9 P f074ef8003bf4826885ed16d0b834c2c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:15.358021 19999 log.cc:824] T 89534a35795349e28a7cc88c2a1c63a9 P 8969eda00dc44d09964a0bad9ca4e813: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:15.358060 19717 heartbeater.cc:503] Master 127.1.182.190:42147 was elected leader, sending a full tablet report...
I20260928 02:09:15.360069 19998 tablet_bootstrap.cc:492] T 89534a35795349e28a7cc88c2a1c63a9 P f074ef8003bf4826885ed16d0b834c2c: No bootstrap required, opened a new log
I20260928 02:09:15.360386 19998 ts_tablet_manager.cc:1397] T 89534a35795349e28a7cc88c2a1c63a9 P f074ef8003bf4826885ed16d0b834c2c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20260928 02:09:15.360504 19997 tablet_bootstrap.cc:492] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336: No bootstrap required, opened a new log
I20260928 02:09:15.360764 19997 ts_tablet_manager.cc:1397] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260928 02:09:15.361166 19999 tablet_bootstrap.cc:492] T 89534a35795349e28a7cc88c2a1c63a9 P 8969eda00dc44d09964a0bad9ca4e813: No bootstrap required, opened a new log
I20260928 02:09:15.361462 19999 ts_tablet_manager.cc:1397] T 89534a35795349e28a7cc88c2a1c63a9 P 8969eda00dc44d09964a0bad9ca4e813: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260928 02:09:15.369217 19997 raft_consensus.cc:348] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7367e54fdb0244ba9481a8903980d336" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 40495 } } peers { permanent_uuid: "f074ef8003bf4826885ed16d0b834c2c" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 41053 } } peers { permanent_uuid: "8969eda00dc44d09964a0bad9ca4e813" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42987 } }
I20260928 02:09:15.369616 19997 raft_consensus.cc:374] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:15.369753 19997 raft_consensus.cc:729] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7367e54fdb0244ba9481a8903980d336, State: Initialized, Role: FOLLOWER
I20260928 02:09:15.370404 19997 consensus_queue.cc:260] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7367e54fdb0244ba9481a8903980d336" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 40495 } } peers { permanent_uuid: "f074ef8003bf4826885ed16d0b834c2c" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 41053 } } peers { permanent_uuid: "8969eda00dc44d09964a0bad9ca4e813" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42987 } }
I20260928 02:09:15.371258 19998 raft_consensus.cc:348] T 89534a35795349e28a7cc88c2a1c63a9 P f074ef8003bf4826885ed16d0b834c2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7367e54fdb0244ba9481a8903980d336" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 40495 } } peers { permanent_uuid: "f074ef8003bf4826885ed16d0b834c2c" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 41053 } } peers { permanent_uuid: "8969eda00dc44d09964a0bad9ca4e813" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42987 } }
I20260928 02:09:15.371752 19998 raft_consensus.cc:374] T 89534a35795349e28a7cc88c2a1c63a9 P f074ef8003bf4826885ed16d0b834c2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:15.371857 19998 raft_consensus.cc:729] T 89534a35795349e28a7cc88c2a1c63a9 P f074ef8003bf4826885ed16d0b834c2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f074ef8003bf4826885ed16d0b834c2c, State: Initialized, Role: FOLLOWER
I20260928 02:09:15.371966 19999 raft_consensus.cc:348] T 89534a35795349e28a7cc88c2a1c63a9 P 8969eda00dc44d09964a0bad9ca4e813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7367e54fdb0244ba9481a8903980d336" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 40495 } } peers { permanent_uuid: "f074ef8003bf4826885ed16d0b834c2c" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 41053 } } peers { permanent_uuid: "8969eda00dc44d09964a0bad9ca4e813" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42987 } }
I20260928 02:09:15.372424 19999 raft_consensus.cc:374] T 89534a35795349e28a7cc88c2a1c63a9 P 8969eda00dc44d09964a0bad9ca4e813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:15.372547 19999 raft_consensus.cc:729] T 89534a35795349e28a7cc88c2a1c63a9 P 8969eda00dc44d09964a0bad9ca4e813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8969eda00dc44d09964a0bad9ca4e813, State: Initialized, Role: FOLLOWER
I20260928 02:09:15.372466 19998 consensus_queue.cc:260] T 89534a35795349e28a7cc88c2a1c63a9 P f074ef8003bf4826885ed16d0b834c2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7367e54fdb0244ba9481a8903980d336" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 40495 } } peers { permanent_uuid: "f074ef8003bf4826885ed16d0b834c2c" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 41053 } } peers { permanent_uuid: "8969eda00dc44d09964a0bad9ca4e813" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42987 } }
I20260928 02:09:15.373292 19999 consensus_queue.cc:260] T 89534a35795349e28a7cc88c2a1c63a9 P 8969eda00dc44d09964a0bad9ca4e813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7367e54fdb0244ba9481a8903980d336" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 40495 } } peers { permanent_uuid: "f074ef8003bf4826885ed16d0b834c2c" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 41053 } } peers { permanent_uuid: "8969eda00dc44d09964a0bad9ca4e813" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42987 } }
I20260928 02:09:15.374693 19998 ts_tablet_manager.cc:1428] T 89534a35795349e28a7cc88c2a1c63a9 P f074ef8003bf4826885ed16d0b834c2c: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
I20260928 02:09:15.375926 19848 heartbeater.cc:503] Master 127.1.182.190:42147 was elected leader, sending a full tablet report...
I20260928 02:09:15.376591 19999 ts_tablet_manager.cc:1428] T 89534a35795349e28a7cc88c2a1c63a9 P 8969eda00dc44d09964a0bad9ca4e813: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260928 02:09:15.378460 19979 heartbeater.cc:503] Master 127.1.182.190:42147 was elected leader, sending a full tablet report...
I20260928 02:09:15.379356 19997 ts_tablet_manager.cc:1428] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336: Time spent starting tablet: real 0.018s user 0.009s sys 0.008s
I20260928 02:09:15.399796 20003 raft_consensus.cc:482] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:09:15.400064 20003 raft_consensus.cc:504] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7367e54fdb0244ba9481a8903980d336" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 40495 } } peers { permanent_uuid: "f074ef8003bf4826885ed16d0b834c2c" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 41053 } } peers { permanent_uuid: "8969eda00dc44d09964a0bad9ca4e813" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42987 } }
I20260928 02:09:15.401809 20003 leader_election.cc:288] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f074ef8003bf4826885ed16d0b834c2c (127.1.182.129:41053), 8969eda00dc44d09964a0bad9ca4e813 (127.1.182.130:42987)
I20260928 02:09:15.408658 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.402368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:34675 (local address 127.1.182.129:41053)
0928 02:09:15.402590 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:15.402596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:15.402618 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:09:15.402700 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:15.402705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:15.402807 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:15.402922 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:15.403558 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.404548 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.406588 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.407758 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.407923 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:15.407995 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:15.408260 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 02:09:15.408389 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 02:09:15.408667 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.402226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:41053 (local address 127.1.182.131:34675)
0928 02:09:15.402410 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:15.402431 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:15.402570 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:15.403086 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:15.403094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:15.403129 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:15.403356 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:15.403368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.404696 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.404701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:15.406407 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:15.406419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.407912 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.407917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:15.408100 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:15.408125 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:09:15.408230 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:09:15.408371 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 02:09:15.409358 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.402725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:51763 (local address 127.1.182.130:42987)
0928 02:09:15.402931 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:15.402939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:15.403525 (+ 586us) server_negotiation.cc:407] Connection header received
0928 02:09:15.403654 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:15.403659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:15.403747 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:15.403850 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:15.404364 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.405479 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.407602 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.408455 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.408635 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:15.408699 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:15.409084 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 02:09:15.409184 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:09:15.409515 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.402584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:42987 (local address 127.1.182.131:51763)
0928 02:09:15.403405 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:15.403423 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:15.403543 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:15.403864 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:15.403867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:15.403885 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:15.404195 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:15.404202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.405648 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.405652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:15.407375 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:15.407391 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.408635 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.408642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:15.408869 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:15.408901 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:09:15.409035 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 02:09:15.409170 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":428,"threads_started":1}
I20260928 02:09:15.410446 19671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89534a35795349e28a7cc88c2a1c63a9" candidate_uuid: "7367e54fdb0244ba9481a8903980d336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f074ef8003bf4826885ed16d0b834c2c" is_pre_election: true
I20260928 02:09:15.410527 19803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89534a35795349e28a7cc88c2a1c63a9" candidate_uuid: "7367e54fdb0244ba9481a8903980d336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8969eda00dc44d09964a0bad9ca4e813" is_pre_election: true
I20260928 02:09:15.410988 19671 raft_consensus.cc:2454] T 89534a35795349e28a7cc88c2a1c63a9 P f074ef8003bf4826885ed16d0b834c2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7367e54fdb0244ba9481a8903980d336 in term 0.
I20260928 02:09:15.411000 19803 raft_consensus.cc:2454] T 89534a35795349e28a7cc88c2a1c63a9 P 8969eda00dc44d09964a0bad9ca4e813 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7367e54fdb0244ba9481a8903980d336 in term 0.
I20260928 02:09:15.411720 19870 leader_election.cc:302] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7367e54fdb0244ba9481a8903980d336, f074ef8003bf4826885ed16d0b834c2c; no voters:
I20260928 02:09:15.412115 20003 raft_consensus.cc:2781] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:09:15.412258 20003 raft_consensus.cc:482] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:09:15.412384 20003 raft_consensus.cc:3037] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:15.414594 20003 raft_consensus.cc:504] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7367e54fdb0244ba9481a8903980d336" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 40495 } } peers { permanent_uuid: "f074ef8003bf4826885ed16d0b834c2c" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 41053 } } peers { permanent_uuid: "8969eda00dc44d09964a0bad9ca4e813" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42987 } }
I20260928 02:09:15.415252 20003 leader_election.cc:288] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 [CANDIDATE]: Term 1 election: Requested vote from peers f074ef8003bf4826885ed16d0b834c2c (127.1.182.129:41053), 8969eda00dc44d09964a0bad9ca4e813 (127.1.182.130:42987)
I20260928 02:09:15.415728 19671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89534a35795349e28a7cc88c2a1c63a9" candidate_uuid: "7367e54fdb0244ba9481a8903980d336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f074ef8003bf4826885ed16d0b834c2c"
I20260928 02:09:15.415789 19803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89534a35795349e28a7cc88c2a1c63a9" candidate_uuid: "7367e54fdb0244ba9481a8903980d336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8969eda00dc44d09964a0bad9ca4e813"
I20260928 02:09:15.415927 19671 raft_consensus.cc:3037] T 89534a35795349e28a7cc88c2a1c63a9 P f074ef8003bf4826885ed16d0b834c2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:15.416009 19803 raft_consensus.cc:3037] T 89534a35795349e28a7cc88c2a1c63a9 P 8969eda00dc44d09964a0bad9ca4e813 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:15.418182 19671 raft_consensus.cc:2454] T 89534a35795349e28a7cc88c2a1c63a9 P f074ef8003bf4826885ed16d0b834c2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7367e54fdb0244ba9481a8903980d336 in term 1.
I20260928 02:09:15.418210 19803 raft_consensus.cc:2454] T 89534a35795349e28a7cc88c2a1c63a9 P 8969eda00dc44d09964a0bad9ca4e813 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7367e54fdb0244ba9481a8903980d336 in term 1.
I20260928 02:09:15.418558 19870 leader_election.cc:302] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7367e54fdb0244ba9481a8903980d336, f074ef8003bf4826885ed16d0b834c2c; no voters:
I20260928 02:09:15.418870 20003 raft_consensus.cc:2781] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:15.419378 20003 raft_consensus.cc:686] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 [term 1 LEADER]: Becoming Leader. State: Replica: 7367e54fdb0244ba9481a8903980d336, State: Running, Role: LEADER
I20260928 02:09:15.419852 20003 consensus_queue.cc:237] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7367e54fdb0244ba9481a8903980d336" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 40495 } } peers { permanent_uuid: "f074ef8003bf4826885ed16d0b834c2c" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 41053 } } peers { permanent_uuid: "8969eda00dc44d09964a0bad9ca4e813" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42987 } }
I20260928 02:09:15.424748 19529 catalog_manager.cc:5495] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7367e54fdb0244ba9481a8903980d336 (127.1.182.131). New cstate: current_term: 1 leader_uuid: "7367e54fdb0244ba9481a8903980d336" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7367e54fdb0244ba9481a8903980d336" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 40495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f074ef8003bf4826885ed16d0b834c2c" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 41053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8969eda00dc44d09964a0bad9ca4e813" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42987 } health_report { overall_health: UNKNOWN } } }
I20260928 02:09:15.460494 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.455158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.1.182.131:40495)
0928 02:09:15.455380 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:15.455386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:15.455437 (+ 51us) server_negotiation.cc:407] Connection header received
0928 02:09:15.455650 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:15.455655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:15.455730 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:15.455853 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:09:15.456137 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.456873 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.458260 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.458695 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.458792 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:15.459405 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:09:15.459932 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:09:15.460056 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:15.460125 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 02:09:15.460189 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:09:15.468816 19671 raft_consensus.cc:1261] T 89534a35795349e28a7cc88c2a1c63a9 P f074ef8003bf4826885ed16d0b834c2c [term 1 FOLLOWER]: Refusing update from remote peer 7367e54fdb0244ba9481a8903980d336: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:09:15.468832 19803 raft_consensus.cc:1261] T 89534a35795349e28a7cc88c2a1c63a9 P 8969eda00dc44d09964a0bad9ca4e813 [term 1 FOLLOWER]: Refusing update from remote peer 7367e54fdb0244ba9481a8903980d336: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:09:15.469805 20007 consensus_queue.cc:1035] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f074ef8003bf4826885ed16d0b834c2c" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 41053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:09:15.470130 20003 consensus_queue.cc:1035] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8969eda00dc44d09964a0bad9ca4e813" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 42987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:09:15.479408 20010 mvcc.cc:205] Tried to move back new op lower bound from 7334139311989780480 to 7334139311803523072. Current Snapshot: MvccSnapshot[applied={T|T < 7334139311989780480}]
I20260928 02:09:15.483925 20011 mvcc.cc:205] Tried to move back new op lower bound from 7334139311989780480 to 7334139311803523072. Current Snapshot: MvccSnapshot[applied={T|T < 7334139311989780480}]
I20260928 02:09:15.507292 19529 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:09:15.507596 19529 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:09:15.511370 19529 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb8bd5c6035a4727bbdf2c4692b4f1de: Sending DeleteTablet for 3 replicas of tablet 89534a35795349e28a7cc88c2a1c63a9
I20260928 02:09:15.512660 19914 tablet_service.cc:1514] Processing DeleteTablet for tablet 89534a35795349e28a7cc88c2a1c63a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:09:15 UTC) from {username='slave'} at 127.0.0.1:48176
I20260928 02:09:15.512715 19651 tablet_service.cc:1514] Processing DeleteTablet for tablet 89534a35795349e28a7cc88c2a1c63a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:09:15 UTC) from {username='slave'} at 127.0.0.1:39588
I20260928 02:09:15.512812 19783 tablet_service.cc:1514] Processing DeleteTablet for tablet 89534a35795349e28a7cc88c2a1c63a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:09:15 UTC) from {username='slave'} at 127.0.0.1:48664
I20260928 02:09:15.519146 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 19589
I20260928 02:09:15.525134 20022 tablet_replica.cc:330] T 89534a35795349e28a7cc88c2a1c63a9 P 8969eda00dc44d09964a0bad9ca4e813: stopping tablet replica
I20260928 02:09:15.525767 20022 raft_consensus.cc:2229] T 89534a35795349e28a7cc88c2a1c63a9 P 8969eda00dc44d09964a0bad9ca4e813 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:09:15.526436 20022 raft_consensus.cc:2258] T 89534a35795349e28a7cc88c2a1c63a9 P 8969eda00dc44d09964a0bad9ca4e813 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:09:15.528179 20020 tablet_replica.cc:330] T 89534a35795349e28a7cc88c2a1c63a9 P f074ef8003bf4826885ed16d0b834c2c: stopping tablet replica
I20260928 02:09:15.528669 20020 raft_consensus.cc:2229] T 89534a35795349e28a7cc88c2a1c63a9 P f074ef8003bf4826885ed16d0b834c2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:09:15.529234 20020 raft_consensus.cc:2258] T 89534a35795349e28a7cc88c2a1c63a9 P f074ef8003bf4826885ed16d0b834c2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:09:15.530125 20021 tablet_replica.cc:330] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336: stopping tablet replica
I20260928 02:09:15.530543 20021 raft_consensus.cc:2229] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:09:15.531281 20021 raft_consensus.cc:2258] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:09:15.532894 20020 ts_tablet_manager.cc:1905] T 89534a35795349e28a7cc88c2a1c63a9 P f074ef8003bf4826885ed16d0b834c2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:09:15.533250 20021 ts_tablet_manager.cc:1905] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:09:15.533365 20022 ts_tablet_manager.cc:1905] T 89534a35795349e28a7cc88c2a1c63a9 P 8969eda00dc44d09964a0bad9ca4e813: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:09:15.536475 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.521383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39592 (local address 127.1.182.129:41053)
0928 02:09:15.521631 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:15.521641 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:15.527379 (+ 5738us) server_negotiation.cc:407] Connection header received
0928 02:09:15.527718 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:15.527724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:15.528000 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:15.528186 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:15.529114 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.530314 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.532462 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.533133 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.533230 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:15.534790 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:15.534803 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:15.534820 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:15.535627 (+ 807us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:15.535684 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:15.535707 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:15.535785 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:15.536015 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 02:09:15.536124 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:09:15.537606 19711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:15.540970 20021 ts_tablet_manager.cc:1918] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:09:15.541213 20021 log.cc:1196] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/89534a35795349e28a7cc88c2a1c63a9
I20260928 02:09:15.541774 20021 ts_tablet_manager.cc:1939] T 89534a35795349e28a7cc88c2a1c63a9 P 7367e54fdb0244ba9481a8903980d336: Deleting consensus metadata
I20260928 02:09:15.543663 19515 catalog_manager.cc:4841] TS 7367e54fdb0244ba9481a8903980d336 (127.1.182.131:40495): tablet 89534a35795349e28a7cc88c2a1c63a9 (table test-table [id=24ad19c19b584e2dbf06516a977c7bbb]) successfully deleted
I20260928 02:09:15.543953 20022 ts_tablet_manager.cc:1918] T 89534a35795349e28a7cc88c2a1c63a9 P 8969eda00dc44d09964a0bad9ca4e813: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:09:15.544687 20022 log.cc:1196] T 89534a35795349e28a7cc88c2a1c63a9 P 8969eda00dc44d09964a0bad9ca4e813: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/89534a35795349e28a7cc88c2a1c63a9
I20260928 02:09:15.545197 20022 ts_tablet_manager.cc:1939] T 89534a35795349e28a7cc88c2a1c63a9 P 8969eda00dc44d09964a0bad9ca4e813: Deleting consensus metadata
I20260928 02:09:15.546640 19516 catalog_manager.cc:4841] TS 8969eda00dc44d09964a0bad9ca4e813 (127.1.182.130:42987): tablet 89534a35795349e28a7cc88c2a1c63a9 (table test-table [id=24ad19c19b584e2dbf06516a977c7bbb]) successfully deleted
I20260928 02:09:15.637112 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 19589
W20260928 02:09:15.655572 19516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.182.129:41053 (error 108)
I20260928 02:09:15.656322 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 19721
I20260928 02:09:15.657812 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.657266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:41053 (local address 127.0.0.1:39608)
0928 02:09:15.657517 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:15.657662 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:41053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260928 02:09:15.658185 19516 catalog_manager.cc:4584] TS f074ef8003bf4826885ed16d0b834c2c (127.1.182.129:41053): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 89534a35795349e28a7cc88c2a1c63a9: Network error: Client connection negotiation failed: client connection to 127.1.182.129:41053: connect: Connection refused (error 111)
I20260928 02:09:15.663599 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.657156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48674 (local address 127.1.182.130:42987)
0928 02:09:15.657339 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:15.657346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:15.657450 (+ 104us) server_negotiation.cc:407] Connection header received
0928 02:09:15.657659 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:15.657665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:15.657776 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:15.657897 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:15.658360 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.659205 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.660552 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.660954 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.661087 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:15.662145 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:15.662154 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:15.662167 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:15.662744 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:15.662790 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:15.662809 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:15.663053 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:15.663213 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 02:09:15.663320 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:09:15.664244 19843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:15.712199 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.711645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:41053 (local address 127.0.0.1:39620)
0928 02:09:15.711894 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:15.712010 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:41053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 02:09:15.713393 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.712975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:41053 (local address 127.0.0.1:39630)
0928 02:09:15.713150 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:15.713241 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:41053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 02:09:15.754859 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 19721
I20260928 02:09:15.774139 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.773509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:41053 (local address 127.0.0.1:39636)
0928 02:09:15.773758 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:15.773900 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:41053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 02:09:15.774546 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 19851
I20260928 02:09:15.775403 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.775021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:41053 (local address 127.0.0.1:39650)
0928 02:09:15.775183 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:15.775261 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:41053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 02:09:15.783243 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.775440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48196 (local address 127.1.182.131:40495)
0928 02:09:15.775640 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:15.775646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:15.776113 (+ 467us) server_negotiation.cc:407] Connection header received
0928 02:09:15.776251 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:15.776257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:15.776391 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:15.776502 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:15.776891 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.778119 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.779653 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:15.780341 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:15.780469 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:15.781531 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:15.781543 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:15.781561 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:15.782394 (+ 833us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:15.782462 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:15.782489 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:15.782599 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:15.782723 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 02:09:15.782830 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:09:15.784041 19974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:15.856194 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.855677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:41053 (local address 127.0.0.1:39664)
0928 02:09:15.855924 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:15.856033 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:41053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260928 02:09:15.857336 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:15.856977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:41053 (local address 127.0.0.1:39678)
0928 02:09:15.857164 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:15.857232 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:41053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260928 02:09:15.905855 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 19851
I20260928 02:09:15.925222 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 19498
I20260928 02:09:15.926409 19559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:16.036870 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 19498
2026-09-28T02:09:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2817 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:09:16 dist-test-slave-23bp krb5kdc[20031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:09:16 dist-test-slave-23bp krb5kdc[20031](info): set up 2 sockets
Sep 28 02:09:16 dist-test-slave-23bp krb5kdc[20031](info): commencing operation
krb5kdc: starting...
W20260928 02:09:17.197664 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:09:17 dist-test-slave-23bp krb5kdc[20031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:09:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:09:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:09:17.454761 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:43273
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:43273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:17.782975 20044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:17.783363 20044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:17.783456 20044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:17.790701 20044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:09:17.790817 20044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:17.790863 20044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:17.790920 20044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:09:17.790966 20044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:09:17.801524 20044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:43273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:43273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:17.802850 20044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:17.804224 20044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:17.816707 20049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:17.816707 20052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:17.816788 20050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:17.817732 20044 server_base.cc:1034] running on GCE node
I20260928 02:09:17.818449 20044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:17.819976 20044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:17.821175 20044 hybrid_clock.cc:648] HybridClock initialized: now 1790561357821110 us; error 50 us; skew 500 ppm
I20260928 02:09:17.821528 20044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:17 dist-test-slave-23bp krb5kdc[20031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561357, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:17.824800 20044 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:09:17.826272 20044 webserver.cc:460] Webserver started at http://127.1.182.190:42289/ using document root <none> and password file <none>
I20260928 02:09:17.826972 20044 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:17.827126 20044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:17.827455 20044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:17.830307 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "59cfd3a4cfc449a4941df8d2336e8419"
format_stamp: "Formatted at 2026-09-28 02:09:17 on dist-test-slave-23bp"
server_key: "56ad077a0aa7afb12923a7d2feab9605"
server_key_iv: "e6d7f246b7510de9a962011b86c4e010"
server_key_version: "encryptionkey@0"
I20260928 02:09:17.831003 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "59cfd3a4cfc449a4941df8d2336e8419"
format_stamp: "Formatted at 2026-09-28 02:09:17 on dist-test-slave-23bp"
server_key: "56ad077a0aa7afb12923a7d2feab9605"
server_key_iv: "e6d7f246b7510de9a962011b86c4e010"
server_key_version: "encryptionkey@0"
I20260928 02:09:17.836129 20044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 02:09:17.839474 20060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:17.840780 20044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 02:09:17.840974 20044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "59cfd3a4cfc449a4941df8d2336e8419"
format_stamp: "Formatted at 2026-09-28 02:09:17 on dist-test-slave-23bp"
server_key: "56ad077a0aa7afb12923a7d2feab9605"
server_key_iv: "e6d7f246b7510de9a962011b86c4e010"
server_key_version: "encryptionkey@0"
I20260928 02:09:17.841182 20044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:17.878122 20044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:17.882668 20044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:17.882957 20044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:17.901631 20044 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:43273
I20260928 02:09:17.901640 20111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:43273 every 8 connection(s)
I20260928 02:09:17.903012 20044 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:09:17.906810 20112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:17.908864 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 20044
I20260928 02:09:17.909066 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:09:17.909600 1754 external_mini_cluster.cc:1428] Setting key 7c872d50208d859b03098df8d481bc2f
I20260928 02:09:17.920310 20112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419: Bootstrap starting.
I20260928 02:09:17.924335 20112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:17.925602 20112 log.cc:824] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:09:17 dist-test-slave-23bp krb5kdc[20031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:09:17.928539 20112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419: No bootstrap required, opened a new log
I20260928 02:09:17.935685 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:17.911671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37364 (local address 127.1.182.190:43273)
0928 02:09:17.912603 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:17.912619 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:17.912669 (+ 50us) server_negotiation.cc:407] Connection header received
0928 02:09:17.914478 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:17.914501 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:17.914923 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:17.915333 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:17.917174 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:17.919020 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:17.920949 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:17.921653 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:17.921801 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:17.926845 (+ 5044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:17.926854 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:17.926870 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:17.926908 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:17.930696 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:17.931591 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:17.931600 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:17.931609 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:17.931831 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:17.932239 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:17.932245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:17.932248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:17.933011 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:17.933324 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:17.933687 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 02:09:17.934183 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":422,"threads_started":1}
I20260928 02:09:17.938527 20112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cfd3a4cfc449a4941df8d2336e8419" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 43273 } }
I20260928 02:09:17.938874 20112 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:17.938951 20112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59cfd3a4cfc449a4941df8d2336e8419, State: Initialized, Role: FOLLOWER
I20260928 02:09:17.939509 20112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cfd3a4cfc449a4941df8d2336e8419" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 43273 } }
I20260928 02:09:17.939723 20112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:09:17.939819 20112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:09:17.939946 20112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:17.942047 20112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cfd3a4cfc449a4941df8d2336e8419" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 43273 } }
I20260928 02:09:17.942556 20112 leader_election.cc:302] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59cfd3a4cfc449a4941df8d2336e8419; no voters:
I20260928 02:09:17.943307 20112 leader_election.cc:288] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:09:17.943420 20117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:17.944188 20117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419 [term 1 LEADER]: Becoming Leader. State: Replica: 59cfd3a4cfc449a4941df8d2336e8419, State: Running, Role: LEADER
I20260928 02:09:17.944819 20117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cfd3a4cfc449a4941df8d2336e8419" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 43273 } }
I20260928 02:09:17.945912 20112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:09:17.947908 20119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59cfd3a4cfc449a4941df8d2336e8419. Latest consensus state: current_term: 1 leader_uuid: "59cfd3a4cfc449a4941df8d2336e8419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cfd3a4cfc449a4941df8d2336e8419" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 43273 } } }
I20260928 02:09:17.948329 20119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:17.949189 20118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59cfd3a4cfc449a4941df8d2336e8419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59cfd3a4cfc449a4941df8d2336e8419" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 43273 } } }
I20260928 02:09:17.949486 20118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:17.953128 20126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:09:17.957639 20126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:09:17.965831 20126 catalog_manager.cc:1347] Generated new cluster ID: fcafae0aa6fa40de9c867efabc63427f
I20260928 02:09:17.965987 20126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:09:17.988068 20126 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:09:17.990002 20126 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:09:18.003162 20126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419: Generated new TSK 0
I20260928 02:09:18.004122 20126 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.182.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
I20260928 02:09:18.062449 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:43273
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:18.392756 20138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:18.393035 20138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:18.393129 20138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:18.400260 20138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:18.400372 20138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:18.400484 20138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:09:18.410140 20138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:43273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:18.411383 20138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:18.412657 20138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:18.425556 20144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:18.425611 20143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:18.425649 20146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:18.426695 20138 server_base.cc:1034] running on GCE node
I20260928 02:09:18.427415 20138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:18.428391 20138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:18.429625 20138 hybrid_clock.cc:648] HybridClock initialized: now 1790561358429577 us; error 43 us; skew 500 ppm
I20260928 02:09:18.429946 20138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:18 dist-test-slave-23bp krb5kdc[20031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561358, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:18.433125 20138 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:09:18.434581 20138 webserver.cc:460] Webserver started at http://127.1.182.129:39961/ using document root <none> and password file <none>
I20260928 02:09:18.435413 20138 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:18.435565 20138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:18.435928 20138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:18.439633 20138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "d098b188f6fb4a25b68d8bd7ae613bab"
format_stamp: "Formatted at 2026-09-28 02:09:18 on dist-test-slave-23bp"
server_key: "0b36f818fc3f5d6320b2e72240bad194"
server_key_iv: "1f9c0b2d91233c995102326abf711d46"
server_key_version: "encryptionkey@0"
I20260928 02:09:18.440346 20138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "d098b188f6fb4a25b68d8bd7ae613bab"
format_stamp: "Formatted at 2026-09-28 02:09:18 on dist-test-slave-23bp"
server_key: "0b36f818fc3f5d6320b2e72240bad194"
server_key_iv: "1f9c0b2d91233c995102326abf711d46"
server_key_version: "encryptionkey@0"
I20260928 02:09:18.445466 20138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:09:18.449254 20153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:18.450789 20138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:09:18.450923 20138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "d098b188f6fb4a25b68d8bd7ae613bab"
format_stamp: "Formatted at 2026-09-28 02:09:18 on dist-test-slave-23bp"
server_key: "0b36f818fc3f5d6320b2e72240bad194"
server_key_iv: "1f9c0b2d91233c995102326abf711d46"
server_key_version: "encryptionkey@0"
I20260928 02:09:18.451151 20138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:18.488094 20138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:18.492168 20138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:18.492470 20138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:18.493464 20138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:09:18.494786 20138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:09:18.494859 20138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:18.495038 20138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:09:18.495165 20138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 02:09:18.528265 20138 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:46403
I20260928 02:09:18.528345 20266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:46403 every 8 connection(s)
I20260928 02:09:18.529592 20138 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:09:18.536861 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 20138
I20260928 02:09:18.537047 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:09:18.537523 1754 external_mini_cluster.cc:1428] Setting key 211cd232d61577490a98cd086a90fbbe
Sep 28 02:09:18 dist-test-slave-23bp krb5kdc[20031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561358, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:09:18.551666 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:18.532098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:51233 (local address 127.1.182.190:43273)
0928 02:09:18.532495 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:18.532499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:18.533562 (+ 1063us) server_negotiation.cc:407] Connection header received
0928 02:09:18.535476 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:18.535481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:18.535567 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:18.535700 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:18.537290 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:18.538095 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:18.539972 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:18.540388 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:18.540525 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:18.544529 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:18.544534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:18.544537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:18.544570 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:18.548103 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:18.548943 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:18.548952 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:18.548958 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:18.549139 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:18.549562 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:18.549569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:18.549575 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:18.549951 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:18.550111 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:18.550893 (+ 782us) server_negotiation.cc:299] Negotiation successful
0928 02:09:18.551197 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":196,"threads_started":1}
I20260928 02:09:18.552330 20270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:18.532696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:43273 (local address 127.1.182.129:51233)
0928 02:09:18.533399 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:18.533447 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:18.535208 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:18.535936 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:18.535943 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:18.536440 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:09:18.537065 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:18.537085 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:18.538244 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:18.538249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:18.539822 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:18.539835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:18.540627 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:18.540634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:18.540871 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:18.541574 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:09:18.541601 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:09:18.544320 (+ 2719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:09:18.548262 (+ 3942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:09:18.548269 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:09:18.548283 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:09:18.548788 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:09:18.549248 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:09:18.549251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:09:18.549253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:09:18.549429 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:09:18.550083 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:09:18.550090 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:09:18.550480 (+ 390us) client_negotiation.cc:769] Sending connection context
0928 02:09:18.550715 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 02:09:18.551196 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":292,"threads_started":1}
I20260928 02:09:18.553927 20267 heartbeater.cc:347] Connected to a master server at 127.1.182.190:43273
I20260928 02:09:18.554299 20267 heartbeater.cc:464] Registering TS with master...
I20260928 02:09:18.555284 20267 heartbeater.cc:511] Master 127.1.182.190:43273 requested a full tablet report, sending...
I20260928 02:09:18.557533 20077 ts_manager.cc:194] Registered new tserver with Master: d098b188f6fb4a25b68d8bd7ae613bab (127.1.182.129:46403)
I20260928 02:09:18.558964 20077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:51233
WARNING: no policy specified for kudu/127.1.182.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
I20260928 02:09:18.594594 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:43273
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:18.921133 20273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:18.921435 20273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:18.921619 20273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:18.928990 20273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:18.929140 20273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:18.929318 20273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:09:18.939220 20273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:43273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:18.940562 20273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:18.941900 20273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:18.956439 20281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:18.956377 20279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:18.956419 20278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:18.957221 20273 server_base.cc:1034] running on GCE node
I20260928 02:09:18.957821 20273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:18.958760 20273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:18.959986 20273 hybrid_clock.cc:648] HybridClock initialized: now 1790561358959941 us; error 56 us; skew 500 ppm
I20260928 02:09:18.960315 20273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:18 dist-test-slave-23bp krb5kdc[20031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561358, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:18.968164 20273 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:09:18.969708 20273 webserver.cc:460] Webserver started at http://127.1.182.130:36665/ using document root <none> and password file <none>
I20260928 02:09:18.970376 20273 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:18.970474 20273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:18.970742 20273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:18.974176 20273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "3ad13b703f1446ee85cd83db2ce83fb5"
format_stamp: "Formatted at 2026-09-28 02:09:18 on dist-test-slave-23bp"
server_key: "f4a3a8accc9d372d8790647d3869cfea"
server_key_iv: "653ecb8b5f6a3e82672c6fbe1e7586a3"
server_key_version: "encryptionkey@0"
I20260928 02:09:18.974905 20273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "3ad13b703f1446ee85cd83db2ce83fb5"
format_stamp: "Formatted at 2026-09-28 02:09:18 on dist-test-slave-23bp"
server_key: "f4a3a8accc9d372d8790647d3869cfea"
server_key_iv: "653ecb8b5f6a3e82672c6fbe1e7586a3"
server_key_version: "encryptionkey@0"
I20260928 02:09:18.980271 20273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 02:09:18.983692 20288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:18.985124 20273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:09:18.985345 20273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "3ad13b703f1446ee85cd83db2ce83fb5"
format_stamp: "Formatted at 2026-09-28 02:09:18 on dist-test-slave-23bp"
server_key: "f4a3a8accc9d372d8790647d3869cfea"
server_key_iv: "653ecb8b5f6a3e82672c6fbe1e7586a3"
server_key_version: "encryptionkey@0"
I20260928 02:09:18.985603 20273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:19.020331 20273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:19.024428 20273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:19.024744 20273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:19.025735 20273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:09:19.027091 20273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:09:19.027169 20273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:19.027333 20273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:09:19.027402 20273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:19.061110 20273 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:37401
I20260928 02:09:19.061218 20401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:37401 every 8 connection(s)
I20260928 02:09:19.062438 20273 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:09:19.071643 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 20273
I20260928 02:09:19.071828 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:09:19.072402 1754 external_mini_cluster.cc:1428] Setting key de898286e6b71d07adba4e571243e5c0
Sep 28 02:09:19 dist-test-slave-23bp krb5kdc[20031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561358, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:09:19.085014 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:19.065077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:38979 (local address 127.1.182.190:43273)
0928 02:09:19.065555 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:19.065561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:19.066559 (+ 998us) server_negotiation.cc:407] Connection header received
0928 02:09:19.068388 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:19.068392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:19.068460 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:19.068565 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:19.070082 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.070811 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.073150 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.073759 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.073880 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:19.077985 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:19.077993 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:19.078000 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:19.078045 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:19.081523 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:19.082531 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:19.082538 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:19.082544 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:19.082706 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:19.083192 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:19.083198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:19.083201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:19.083474 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:19.083636 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:19.084404 (+ 768us) server_negotiation.cc:299] Negotiation successful
0928 02:09:19.084634 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":227,"threads_started":1}
I20260928 02:09:19.086037 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:19.065681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:43273 (local address 127.1.182.130:38979)
0928 02:09:19.066388 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:19.066434 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:19.068135 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:19.068785 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:19.068793 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:19.069304 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:09:19.069882 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:19.069899 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.070949 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.070952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:19.072999 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:19.073018 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.073975 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.073980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:19.074178 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:19.074938 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:09:19.074961 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:09:19.077725 (+ 2764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:09:19.081681 (+ 3956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:09:19.081694 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:09:19.081716 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:09:19.082397 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:09:19.082834 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:09:19.082840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:09:19.082844 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:09:19.083040 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:09:19.083613 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:09:19.083622 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:09:19.083975 (+ 353us) client_negotiation.cc:769] Sending connection context
0928 02:09:19.084298 (+ 323us) client_negotiation.cc:240] Negotiation successful
0928 02:09:19.084870 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":285,"threads_started":1}
I20260928 02:09:19.087656 20402 heartbeater.cc:347] Connected to a master server at 127.1.182.190:43273
I20260928 02:09:19.087991 20402 heartbeater.cc:464] Registering TS with master...
I20260928 02:09:19.088907 20402 heartbeater.cc:511] Master 127.1.182.190:43273 requested a full tablet report, sending...
I20260928 02:09:19.090586 20077 ts_manager.cc:194] Registered new tserver with Master: 3ad13b703f1446ee85cd83db2ce83fb5 (127.1.182.130:37401)
I20260928 02:09:19.091518 20077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:38979
WARNING: no policy specified for kudu/127.1.182.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
I20260928 02:09:19.131141 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:43273
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:19.464915 20408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:19.465226 20408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:19.465323 20408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:19.472609 20408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:19.472741 20408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:19.472854 20408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:09:19.482928 20408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:43273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:19.484405 20408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:19.485741 20408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:19.499279 20413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:19.499311 20416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:19.499413 20414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:19.500454 20408 server_base.cc:1034] running on GCE node
I20260928 02:09:19.501050 20408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:19.502000 20408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:19.503226 20408 hybrid_clock.cc:648] HybridClock initialized: now 1790561359503181 us; error 45 us; skew 500 ppm
I20260928 02:09:19.503561 20408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:19 dist-test-slave-23bp krb5kdc[20031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561359, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:19.506695 20408 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:09:19.508219 20408 webserver.cc:460] Webserver started at http://127.1.182.131:38095/ using document root <none> and password file <none>
I20260928 02:09:19.508932 20408 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:19.509078 20408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:19.509382 20408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:19.512423 20408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "acc1884c378347c0bc6a6e69e7675475"
format_stamp: "Formatted at 2026-09-28 02:09:19 on dist-test-slave-23bp"
server_key: "b9d188999d09c7cf0373608abb915f7c"
server_key_iv: "f61296df5fc07db90bab4b6908cd9b10"
server_key_version: "encryptionkey@0"
I20260928 02:09:19.513216 20408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "acc1884c378347c0bc6a6e69e7675475"
format_stamp: "Formatted at 2026-09-28 02:09:19 on dist-test-slave-23bp"
server_key: "b9d188999d09c7cf0373608abb915f7c"
server_key_iv: "f61296df5fc07db90bab4b6908cd9b10"
server_key_version: "encryptionkey@0"
I20260928 02:09:19.518317 20408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:09:19.521821 20423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:19.523180 20408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:09:19.523346 20408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "acc1884c378347c0bc6a6e69e7675475"
format_stamp: "Formatted at 2026-09-28 02:09:19 on dist-test-slave-23bp"
server_key: "b9d188999d09c7cf0373608abb915f7c"
server_key_iv: "f61296df5fc07db90bab4b6908cd9b10"
server_key_version: "encryptionkey@0"
I20260928 02:09:19.523554 20408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:19.547205 20408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:19.551888 20408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:19.552184 20408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:19.553177 20408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:09:19.554531 20408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:09:19.554636 20408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:19.554773 20408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:09:19.554842 20408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:19.563403 20267 heartbeater.cc:503] Master 127.1.182.190:43273 was elected leader, sending a full tablet report...
I20260928 02:09:19.589229 20408 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:37647
I20260928 02:09:19.589354 20537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:37647 every 8 connection(s)
I20260928 02:09:19.590663 20408 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:09:19.595867 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 20408
I20260928 02:09:19.596051 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:09:19.596504 1754 external_mini_cluster.cc:1428] Setting key 93fba2b3b723ede529594aa091bb7556
Sep 28 02:09:19 dist-test-slave-23bp krb5kdc[20031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561359, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:09:19.610419 20540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:19.593228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:43267 (local address 127.1.182.190:43273)
0928 02:09:19.593673 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:19.593678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:19.594698 (+ 1020us) server_negotiation.cc:407] Connection header received
0928 02:09:19.596764 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:19.596771 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:19.596868 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:19.596991 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:19.598755 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.599790 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.601272 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.601648 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.601738 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:19.605003 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:19.605008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:19.605011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:19.605043 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:19.607599 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:19.608312 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:19.608316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:19.608318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:19.608430 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:19.608794 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:19.608797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:19.608799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:19.609054 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:19.609175 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:19.609842 (+ 667us) server_negotiation.cc:299] Negotiation successful
0928 02:09:19.610079 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":211,"threads_started":1}
I20260928 02:09:19.611613 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:19.593851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:43273 (local address 127.1.182.131:43267)
0928 02:09:19.594537 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:19.594574 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:19.596428 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:19.597296 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:19.597305 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:19.597919 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:09:19.598599 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:19.598617 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.599931 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.599935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:19.601154 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:19.601165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.601828 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.601832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:19.602017 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:19.602628 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:09:19.602654 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:09:19.604803 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:09:19.607741 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:09:19.607749 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:09:19.607763 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:09:19.608210 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:09:19.608524 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:09:19.608527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:09:19.608530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:09:19.608704 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:09:19.609175 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:09:19.609180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:09:19.609493 (+ 313us) client_negotiation.cc:769] Sending connection context
0928 02:09:19.609734 (+ 241us) client_negotiation.cc:240] Negotiation successful
0928 02:09:19.610278 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":270,"threads_started":1}
I20260928 02:09:19.612951 20538 heartbeater.cc:347] Connected to a master server at 127.1.182.190:43273
I20260928 02:09:19.613261 20538 heartbeater.cc:464] Registering TS with master...
I20260928 02:09:19.613948 20538 heartbeater.cc:511] Master 127.1.182.190:43273 requested a full tablet report, sending...
I20260928 02:09:19.615396 20077 ts_manager.cc:194] Registered new tserver with Master: acc1884c378347c0bc6a6e69e7675475 (127.1.182.131:37647)
I20260928 02:09:19.616147 20077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:43267
I20260928 02:09:19.624198 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:09:19.640959 20540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:19.628419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.1.182.190:43273)
0928 02:09:19.628614 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:19.628618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:19.629084 (+ 466us) server_negotiation.cc:407] Connection header received
0928 02:09:19.629224 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:19.629228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:19.629288 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:19.629364 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:19.630808 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.631584 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.632817 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.633222 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.633317 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:19.635852 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:19.635857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:19.635860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:19.635892 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:19.638522 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:19.639315 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:19.639318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:19.639321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:19.639431 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:19.639804 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:19.639807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:19.639809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:19.640093 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:19.640277 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:19.640598 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 02:09:19.640733 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:09:19.644836 20077 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:09:19.652190 20077 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:09:19.694036 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:19.681515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:37401 (local address 127.0.0.1:41192)
0928 02:09:19.685768 (+ 4253us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:19.685791 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:19.685899 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:19.687112 (+ 1213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:19.687118 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:19.687147 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:19.687591 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:19.687604 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.689548 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.689554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:19.691392 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:19.691404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.693111 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.693118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:19.693320 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:19.693368 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 02:09:19.693487 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 02:09:19.693689 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4147,"mutex_wait_us":52,"thread_start_us":3234,"threads_started":1}
I20260928 02:09:19.694846 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:19.681766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.1.182.130:37401)
0928 02:09:19.686519 (+ 4753us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:19.686527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:19.686552 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:09:19.686646 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:19.686655 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:19.686866 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:19.687802 (+ 936us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:19.688020 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.689381 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.691533 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.692897 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.693056 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:19.693274 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:19.694349 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0928 02:09:19.694524 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4589,"thread_start_us":4484,"threads_started":1}
I20260928 02:09:19.695048 20552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:19.664015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:46403 (local address 127.0.0.1:34304)
0928 02:09:19.679844 (+ 15829us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:19.679917 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:19.680156 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:19.686267 (+ 6111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:19.686276 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:19.686309 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:19.686791 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:19.686812 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.689125 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.689130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:19.691750 (+ 2620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:19.691765 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.693882 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.693888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:19.694075 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:19.694109 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:09:19.694459 (+ 350us) client_negotiation.cc:240] Negotiation successful
0928 02:09:19.694600 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15424,"mutex_wait_us":78,"thread_start_us":21982,"threads_started":1}
I20260928 02:09:19.695892 20553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:19.664271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34304 (local address 127.1.182.129:46403)
0928 02:09:19.685533 (+ 21262us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:19.685546 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:19.685583 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:09:19.685694 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:19.685703 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:19.686010 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:19.687043 (+ 1033us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:19.687300 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.688976 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.691916 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.693350 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.693503 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:19.693732 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:19.695406 (+ 1674us) server_negotiation.cc:299] Negotiation successful
0928 02:09:19.695536 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21060,"thread_start_us":20898,"threads_started":1}
I20260928 02:09:19.696134 20555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:19.664227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:37647 (local address 127.0.0.1:45746)
0928 02:09:19.685121 (+ 20894us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:19.685149 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:19.685312 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:19.686528 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:19.686532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:19.686560 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:19.686958 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:19.686972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.688891 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.688899 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:19.691750 (+ 2851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:19.691765 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.695269 (+ 3504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.695279 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:19.695466 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:19.695495 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 02:09:19.695596 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 02:09:19.695720 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20715,"mutex_wait_us":133,"thread_start_us":16170,"threads_started":1}
I20260928 02:09:19.696184 20554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:19.664951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45746 (local address 127.1.182.131:37647)
0928 02:09:19.680250 (+ 15299us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:19.680264 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:19.685492 (+ 5228us) server_negotiation.cc:407] Connection header received
0928 02:09:19.685656 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:19.685668 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:19.686017 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:19.686249 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:19.687216 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.688769 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.693381 (+ 4612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.694900 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.695045 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:19.695298 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:19.695670 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 02:09:19.695830 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15050,"thread_start_us":15324,"threads_started":1}
I20260928 02:09:19.699361 20472 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3cdf2190e44c0fa081714743d8093f (DEFAULT_TABLE table=test-table [id=013ac6b8e5fd4363999ea798e3d3bddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:09:19.699466 20201 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3cdf2190e44c0fa081714743d8093f (DEFAULT_TABLE table=test-table [id=013ac6b8e5fd4363999ea798e3d3bddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:09:19.699466 20337 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3cdf2190e44c0fa081714743d8093f (DEFAULT_TABLE table=test-table [id=013ac6b8e5fd4363999ea798e3d3bddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:09:19.700690 20472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b3cdf2190e44c0fa081714743d8093f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:19.700798 20201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b3cdf2190e44c0fa081714743d8093f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:19.700815 20337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b3cdf2190e44c0fa081714743d8093f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:19.712699 20557 tablet_bootstrap.cc:492] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475: Bootstrap starting.
I20260928 02:09:19.716379 20558 tablet_bootstrap.cc:492] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5: Bootstrap starting.
I20260928 02:09:19.717031 20557 tablet_bootstrap.cc:654] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:19.718101 20559 tablet_bootstrap.cc:492] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab: Bootstrap starting.
I20260928 02:09:19.718516 20557 log.cc:824] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:19.719854 20558 tablet_bootstrap.cc:654] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:19.721180 20558 log.cc:824] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:19.721481 20559 tablet_bootstrap.cc:654] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:19.722016 20557 tablet_bootstrap.cc:492] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475: No bootstrap required, opened a new log
I20260928 02:09:19.722285 20557 ts_tablet_manager.cc:1397] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20260928 02:09:19.722697 20559 log.cc:824] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:19.724259 20558 tablet_bootstrap.cc:492] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5: No bootstrap required, opened a new log
I20260928 02:09:19.724509 20558 ts_tablet_manager.cc:1397] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260928 02:09:19.725229 20559 tablet_bootstrap.cc:492] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab: No bootstrap required, opened a new log
I20260928 02:09:19.725510 20559 ts_tablet_manager.cc:1397] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 02:09:19.733206 20559 raft_consensus.cc:348] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d098b188f6fb4a25b68d8bd7ae613bab" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46403 } } peers { permanent_uuid: "acc1884c378347c0bc6a6e69e7675475" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37647 } } peers { permanent_uuid: "3ad13b703f1446ee85cd83db2ce83fb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37401 } }
I20260928 02:09:19.733145 20557 raft_consensus.cc:348] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d098b188f6fb4a25b68d8bd7ae613bab" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46403 } } peers { permanent_uuid: "acc1884c378347c0bc6a6e69e7675475" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37647 } } peers { permanent_uuid: "3ad13b703f1446ee85cd83db2ce83fb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37401 } }
I20260928 02:09:19.733577 20559 raft_consensus.cc:374] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:19.733594 20557 raft_consensus.cc:374] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:19.733672 20559 raft_consensus.cc:729] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d098b188f6fb4a25b68d8bd7ae613bab, State: Initialized, Role: FOLLOWER
I20260928 02:09:19.733691 20557 raft_consensus.cc:729] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc1884c378347c0bc6a6e69e7675475, State: Initialized, Role: FOLLOWER
I20260928 02:09:19.734320 20559 consensus_queue.cc:260] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d098b188f6fb4a25b68d8bd7ae613bab" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46403 } } peers { permanent_uuid: "acc1884c378347c0bc6a6e69e7675475" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37647 } } peers { permanent_uuid: "3ad13b703f1446ee85cd83db2ce83fb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37401 } }
I20260928 02:09:19.734341 20557 consensus_queue.cc:260] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d098b188f6fb4a25b68d8bd7ae613bab" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46403 } } peers { permanent_uuid: "acc1884c378347c0bc6a6e69e7675475" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37647 } } peers { permanent_uuid: "3ad13b703f1446ee85cd83db2ce83fb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37401 } }
I20260928 02:09:19.735935 20538 heartbeater.cc:503] Master 127.1.182.190:43273 was elected leader, sending a full tablet report...
I20260928 02:09:19.736209 20559 ts_tablet_manager.cc:1428] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260928 02:09:19.736698 20558 raft_consensus.cc:348] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d098b188f6fb4a25b68d8bd7ae613bab" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46403 } } peers { permanent_uuid: "acc1884c378347c0bc6a6e69e7675475" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37647 } } peers { permanent_uuid: "3ad13b703f1446ee85cd83db2ce83fb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37401 } }
I20260928 02:09:19.737164 20558 raft_consensus.cc:374] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:19.737270 20558 raft_consensus.cc:729] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad13b703f1446ee85cd83db2ce83fb5, State: Initialized, Role: FOLLOWER
I20260928 02:09:19.737710 20557 ts_tablet_manager.cc:1428] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
I20260928 02:09:19.737879 20558 consensus_queue.cc:260] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d098b188f6fb4a25b68d8bd7ae613bab" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46403 } } peers { permanent_uuid: "acc1884c378347c0bc6a6e69e7675475" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37647 } } peers { permanent_uuid: "3ad13b703f1446ee85cd83db2ce83fb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37401 } }
I20260928 02:09:19.738986 20402 heartbeater.cc:503] Master 127.1.182.190:43273 was elected leader, sending a full tablet report...
I20260928 02:09:19.739463 20558 ts_tablet_manager.cc:1428] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5: Time spent starting tablet: real 0.015s user 0.006s sys 0.009s
W20260928 02:09:19.782301 20268 tablet.cc:2369] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:09:19.814582 20403 tablet.cc:2369] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:09:19.842343 20539 tablet.cc:2369] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:09:19.941118 20564 raft_consensus.cc:482] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:09:19.941402 20564 raft_consensus.cc:504] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d098b188f6fb4a25b68d8bd7ae613bab" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46403 } } peers { permanent_uuid: "acc1884c378347c0bc6a6e69e7675475" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37647 } } peers { permanent_uuid: "3ad13b703f1446ee85cd83db2ce83fb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37401 } }
I20260928 02:09:19.943199 20564 leader_election.cc:288] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d098b188f6fb4a25b68d8bd7ae613bab (127.1.182.129:46403), 3ad13b703f1446ee85cd83db2ce83fb5 (127.1.182.130:37401)
I20260928 02:09:19.950281 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:19.943754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:53183 (local address 127.1.182.130:37401)
0928 02:09:19.943958 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:19.943966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:19.944249 (+ 283us) server_negotiation.cc:407] Connection header received
0928 02:09:19.944493 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:19.944500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:19.944609 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:19.944682 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:19.945241 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.946209 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.948278 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.949283 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.949408 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:19.949462 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:19.949836 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 02:09:19.949978 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:09:19.950563 20553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:19.943702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:53575 (local address 127.1.182.129:46403)
0928 02:09:19.943943 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:19.943949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:19.944797 (+ 848us) server_negotiation.cc:407] Connection header received
0928 02:09:19.944938 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:19.944942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:19.944999 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:19.945072 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:19.945687 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.946636 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.948576 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.949595 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.949756 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:19.949834 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:19.950199 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 02:09:19.950328 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 02:09:19.950591 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:19.943745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:37401 (local address 127.1.182.131:53183)
0928 02:09:19.944094 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:19.944133 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:19.944299 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:19.944730 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:19.944738 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:19.944772 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:19.945034 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:19.945069 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.946379 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.946385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:19.948123 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:19.948135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.949433 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.949440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:19.949653 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:19.949697 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 02:09:19.949797 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 02:09:19.949937 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":41,"spinlock_wait_cycles":27008}
I20260928 02:09:19.950819 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:19.943833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:46403 (local address 127.1.182.131:53575)
0928 02:09:19.944680 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:19.944697 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:19.944806 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:19.945115 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:19.945123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:19.945151 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:19.945523 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:19.945534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.946759 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.946762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:19.948420 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:19.948435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:19.949807 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:19.949813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:19.950013 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:19.950043 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 02:09:19.950138 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 02:09:19.950260 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":19,"thread_start_us":419,"threads_started":1}
I20260928 02:09:19.952023 20222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3cdf2190e44c0fa081714743d8093f" candidate_uuid: "acc1884c378347c0bc6a6e69e7675475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d098b188f6fb4a25b68d8bd7ae613bab" is_pre_election: true
I20260928 02:09:19.952023 20357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3cdf2190e44c0fa081714743d8093f" candidate_uuid: "acc1884c378347c0bc6a6e69e7675475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad13b703f1446ee85cd83db2ce83fb5" is_pre_election: true
I20260928 02:09:19.952632 20357 raft_consensus.cc:2454] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc1884c378347c0bc6a6e69e7675475 in term 0.
I20260928 02:09:19.952632 20222 raft_consensus.cc:2454] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc1884c378347c0bc6a6e69e7675475 in term 0.
I20260928 02:09:19.953418 20424 leader_election.cc:302] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ad13b703f1446ee85cd83db2ce83fb5, acc1884c378347c0bc6a6e69e7675475; no voters:
I20260928 02:09:19.953778 20564 raft_consensus.cc:2781] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:09:19.953948 20564 raft_consensus.cc:482] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:09:19.954121 20564 raft_consensus.cc:3037] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:19.956488 20564 raft_consensus.cc:504] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d098b188f6fb4a25b68d8bd7ae613bab" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46403 } } peers { permanent_uuid: "acc1884c378347c0bc6a6e69e7675475" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37647 } } peers { permanent_uuid: "3ad13b703f1446ee85cd83db2ce83fb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37401 } }
I20260928 02:09:19.957144 20564 leader_election.cc:288] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 [CANDIDATE]: Term 1 election: Requested vote from peers d098b188f6fb4a25b68d8bd7ae613bab (127.1.182.129:46403), 3ad13b703f1446ee85cd83db2ce83fb5 (127.1.182.130:37401)
I20260928 02:09:19.957571 20222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3cdf2190e44c0fa081714743d8093f" candidate_uuid: "acc1884c378347c0bc6a6e69e7675475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d098b188f6fb4a25b68d8bd7ae613bab"
I20260928 02:09:19.957585 20357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3cdf2190e44c0fa081714743d8093f" candidate_uuid: "acc1884c378347c0bc6a6e69e7675475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad13b703f1446ee85cd83db2ce83fb5"
I20260928 02:09:19.957789 20222 raft_consensus.cc:3037] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:19.957787 20357 raft_consensus.cc:3037] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:19.960522 20222 raft_consensus.cc:2454] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc1884c378347c0bc6a6e69e7675475 in term 1.
I20260928 02:09:19.960531 20357 raft_consensus.cc:2454] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc1884c378347c0bc6a6e69e7675475 in term 1.
I20260928 02:09:19.960951 20424 leader_election.cc:302] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ad13b703f1446ee85cd83db2ce83fb5, acc1884c378347c0bc6a6e69e7675475; no voters:
I20260928 02:09:19.961248 20564 raft_consensus.cc:2781] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:19.961719 20564 raft_consensus.cc:686] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 [term 1 LEADER]: Becoming Leader. State: Replica: acc1884c378347c0bc6a6e69e7675475, State: Running, Role: LEADER
I20260928 02:09:19.962296 20564 consensus_queue.cc:237] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d098b188f6fb4a25b68d8bd7ae613bab" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46403 } } peers { permanent_uuid: "acc1884c378347c0bc6a6e69e7675475" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37647 } } peers { permanent_uuid: "3ad13b703f1446ee85cd83db2ce83fb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37401 } }
I20260928 02:09:19.967180 20077 catalog_manager.cc:5495] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 reported cstate change: term changed from 0 to 1, leader changed from <none> to acc1884c378347c0bc6a6e69e7675475 (127.1.182.131). New cstate: current_term: 1 leader_uuid: "acc1884c378347c0bc6a6e69e7675475" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d098b188f6fb4a25b68d8bd7ae613bab" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc1884c378347c0bc6a6e69e7675475" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ad13b703f1446ee85cd83db2ce83fb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37401 } health_report { overall_health: UNKNOWN } } }
I20260928 02:09:20.042259 20554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:20.037137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45760 (local address 127.1.182.131:37647)
0928 02:09:20.037304 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:20.037309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:20.037426 (+ 117us) server_negotiation.cc:407] Connection header received
0928 02:09:20.037612 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:20.037616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:20.037695 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:20.037827 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:09:20.038226 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:20.038917 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:20.040254 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:20.040661 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:20.040756 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:20.041267 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:09:20.041766 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:09:20.041896 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:20.042004 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 02:09:20.042044 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:09:20.050729 20357 raft_consensus.cc:1261] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5 [term 1 FOLLOWER]: Refusing update from remote peer acc1884c378347c0bc6a6e69e7675475: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:09:20.050732 20222 raft_consensus.cc:1261] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab [term 1 FOLLOWER]: Refusing update from remote peer acc1884c378347c0bc6a6e69e7675475: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:09:20.051807 20564 consensus_queue.cc:1035] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d098b188f6fb4a25b68d8bd7ae613bab" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 46403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:09:20.052103 20567 consensus_queue.cc:1035] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ad13b703f1446ee85cd83db2ce83fb5" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:09:20.064116 20570 mvcc.cc:205] Tried to move back new op lower bound from 7334139330756952064 to 7334139330409697280. Current Snapshot: MvccSnapshot[applied={T|T < 7334139330756952064}]
I20260928 02:09:20.090199 20077 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:09:20.090467 20077 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:09:20.095134 20077 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59cfd3a4cfc449a4941df8d2336e8419: Sending DeleteTablet for 3 replicas of tablet 2b3cdf2190e44c0fa081714743d8093f
I20260928 02:09:20.096288 20201 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b3cdf2190e44c0fa081714743d8093f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:09:20 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:34304
I20260928 02:09:20.096817 20337 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b3cdf2190e44c0fa081714743d8093f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:09:20 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:41192
I20260928 02:09:20.097028 20472 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b3cdf2190e44c0fa081714743d8093f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:09:20 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:45746
I20260928 02:09:20.097252 20471 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b3cdf2190e44c0fa081714743d8093f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:09:20 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:45746
I20260928 02:09:20.099659 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 20138
I20260928 02:09:20.107661 20582 tablet_replica.cc:330] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475: stopping tablet replica
I20260928 02:09:20.108276 20582 raft_consensus.cc:2229] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:09:20.109110 20582 raft_consensus.cc:2258] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:09:20.110088 20580 tablet_replica.cc:330] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab: stopping tablet replica
I20260928 02:09:20.110597 20580 raft_consensus.cc:2229] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:09:20.111279 20580 raft_consensus.cc:2258] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:09:20.112200 20581 tablet_replica.cc:330] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5: stopping tablet replica
I20260928 02:09:20.112733 20581 raft_consensus.cc:2229] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:09:20.113199 20581 raft_consensus.cc:2258] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:09:20.113967 20580 ts_tablet_manager.cc:1905] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 02:09:20 dist-test-slave-23bp krb5kdc[20031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:09:20.120697 20582 ts_tablet_manager.cc:1905] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:09:20.121501 20580 ts_tablet_manager.cc:1918] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:09:20.121665 20580 log.cc:1196] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-0/wal/wals/2b3cdf2190e44c0fa081714743d8093f
I20260928 02:09:20.122229 20580 ts_tablet_manager.cc:1939] T 2b3cdf2190e44c0fa081714743d8093f P d098b188f6fb4a25b68d8bd7ae613bab: Deleting consensus metadata
I20260928 02:09:20.123188 20581 ts_tablet_manager.cc:1905] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:09:20.123347 20553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:20.101091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34320 (local address 127.1.182.129:46403)
0928 02:09:20.101298 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:20.101307 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:20.102196 (+ 889us) server_negotiation.cc:407] Connection header received
0928 02:09:20.102498 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:20.102506 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:20.102610 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:20.102691 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:20.104793 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:20.106224 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:20.108628 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:20.109598 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:20.109753 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:20.115492 (+ 5739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:20.115506 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:20.115522 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:20.115577 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:20.119673 (+ 4096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:20.120561 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:20.120570 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:20.120574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:20.120735 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:20.121222 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:20.121230 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:20.121241 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:20.121723 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:20.122071 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:20.122666 (+ 595us) server_negotiation.cc:299] Negotiation successful
0928 02:09:20.122908 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:09:20.124333 20063 catalog_manager.cc:4841] TS d098b188f6fb4a25b68d8bd7ae613bab (127.1.182.129:46403): tablet 2b3cdf2190e44c0fa081714743d8093f (table test-table [id=013ac6b8e5fd4363999ea798e3d3bddb]) successfully deleted
I20260928 02:09:20.124821 20262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:20.137262 20582 ts_tablet_manager.cc:1918] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:09:20.137481 20582 log.cc:1196] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/2b3cdf2190e44c0fa081714743d8093f
I20260928 02:09:20.138103 20582 ts_tablet_manager.cc:1939] T 2b3cdf2190e44c0fa081714743d8093f P acc1884c378347c0bc6a6e69e7675475: Deleting consensus metadata
I20260928 02:09:20.140347 20061 catalog_manager.cc:4841] TS acc1884c378347c0bc6a6e69e7675475 (127.1.182.131:37647): tablet 2b3cdf2190e44c0fa081714743d8093f (table test-table [id=013ac6b8e5fd4363999ea798e3d3bddb]) successfully deleted
W20260928 02:09:20.141185 20061 catalog_manager.cc:4808] TS acc1884c378347c0bc6a6e69e7675475 (127.1.182.131:37647): delete failed for tablet 2b3cdf2190e44c0fa081714743d8093f because the tablet was not found. No further retry: Not found: Tablet not found: 2b3cdf2190e44c0fa081714743d8093f
I20260928 02:09:20.141983 20581 ts_tablet_manager.cc:1918] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:09:20.142201 20581 log.cc:1196] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/2b3cdf2190e44c0fa081714743d8093f
I20260928 02:09:20.142674 20581 ts_tablet_manager.cc:1939] T 2b3cdf2190e44c0fa081714743d8093f P 3ad13b703f1446ee85cd83db2ce83fb5: Deleting consensus metadata
I20260928 02:09:20.144248 20061 catalog_manager.cc:4841] TS 3ad13b703f1446ee85cd83db2ce83fb5 (127.1.182.130:37401): tablet 2b3cdf2190e44c0fa081714743d8093f (table test-table [id=013ac6b8e5fd4363999ea798e3d3bddb]) successfully deleted
I20260928 02:09:20.284204 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 20138
I20260928 02:09:20.308065 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 20273
Sep 28 02:09:20 dist-test-slave-23bp krb5kdc[20031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.130@KRBTEST.COM
I20260928 02:09:20.325691 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:20.308968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.1.182.130:37401)
0928 02:09:20.309144 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:20.309148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:20.309166 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:09:20.309380 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:20.309384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:20.309448 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:20.309530 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:20.311234 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:20.311932 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:20.313765 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:20.314140 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:20.314236 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:20.319770 (+ 5534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:20.319786 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:20.319819 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:20.319859 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:20.322712 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:20.323640 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:20.323648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:20.323650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:20.323784 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:20.324204 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:20.324208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:20.324210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:20.324592 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:20.324811 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:20.325206 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 02:09:20.325371 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:09:20.326390 20397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:20.417677 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 20273
I20260928 02:09:20.441464 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 20408
Sep 28 02:09:20 dist-test-slave-23bp krb5kdc[20031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.131@KRBTEST.COM
I20260928 02:09:20.460404 20554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:20.442302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.1.182.131:37647)
0928 02:09:20.442657 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:20.442696 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:20.442719 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:09:20.442889 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:20.442894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:20.443014 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:20.443169 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:20.445191 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:20.446573 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:20.448165 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:20.448894 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:20.449039 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:20.454268 (+ 5229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:20.454281 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:20.454296 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:20.454339 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:20.457417 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:20.458270 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:20.458277 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:20.458279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:20.458444 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:20.458852 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:20.458856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:20.458858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:20.459314 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:20.459527 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:20.459893 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 02:09:20.460056 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260928 02:09:20.461135 20533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:20.583401 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 20408
I20260928 02:09:20.602823 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 20044
I20260928 02:09:20.604091 20107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:20.716019 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 20044
2026-09-28T02:09:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4681 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T02:09:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:09:20Z Disabled control of system clock
I20260928 02:09:20.795343 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:37105
--webserver_interface=127.1.182.190
--webserver_port=0
--builtin_ntp_servers=127.1.182.148:36629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:37105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:09:21.136376 20591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:21.136713 20591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:21.136802 20591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:21.144856 20591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:09:21.145008 20591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:21.145057 20591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:21.145115 20591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:09:21.145160 20591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:09:21.156186 20591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:36629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:37105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:37105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:21.158267 20591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:21.160687 20591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:21.173326 20597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:21.173336 20599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:21.173411 20596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:21.174134 20591 server_base.cc:1034] running on GCE node
I20260928 02:09:21.174907 20591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:21.176427 20591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:21.177662 20591 hybrid_clock.cc:648] HybridClock initialized: now 1790561361177615 us; error 45 us; skew 500 ppm
I20260928 02:09:21.178028 20591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:09:21.180289 20591 webserver.cc:460] Webserver started at http://127.1.182.190:44247/ using document root <none> and password file <none>
I20260928 02:09:21.181056 20591 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:21.181185 20591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:21.181504 20591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:21.184549 20591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "67aa46ddcfd749a1b89a54c38f033a18"
format_stamp: "Formatted at 2026-09-28 02:09:21 on dist-test-slave-23bp"
server_key: "ea0b09f6a43cb594fc862a445ca7d2ba"
server_key_iv: "7a79d899b103905eb74bf79706426401"
server_key_version: "encryptionkey@0"
I20260928 02:09:21.185365 20591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "67aa46ddcfd749a1b89a54c38f033a18"
format_stamp: "Formatted at 2026-09-28 02:09:21 on dist-test-slave-23bp"
server_key: "ea0b09f6a43cb594fc862a445ca7d2ba"
server_key_iv: "7a79d899b103905eb74bf79706426401"
server_key_version: "encryptionkey@0"
I20260928 02:09:21.190570 20591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:09:21.194031 20605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:21.195637 20591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 02:09:21.195837 20591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "67aa46ddcfd749a1b89a54c38f033a18"
format_stamp: "Formatted at 2026-09-28 02:09:21 on dist-test-slave-23bp"
server_key: "ea0b09f6a43cb594fc862a445ca7d2ba"
server_key_iv: "7a79d899b103905eb74bf79706426401"
server_key_version: "encryptionkey@0"
I20260928 02:09:21.196070 20591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:21.217075 20591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:21.218001 20591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:21.218338 20591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:21.237545 20591 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:37105
I20260928 02:09:21.237568 20656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:37105 every 8 connection(s)
I20260928 02:09:21.239027 20591 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:09:21.241110 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 20591
I20260928 02:09:21.241313 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:09:21.241868 1754 external_mini_cluster.cc:1428] Setting key c02123dc8e169fbed6ac006e768df890
I20260928 02:09:21.243907 20657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:21.256498 20661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:21.243984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46056 (local address 127.1.182.190:37105)
0928 02:09:21.245060 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:21.245077 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:21.245117 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:09:21.246718 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:21.246739 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:21.247305 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:21.247660 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:21.248281 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:21.249664 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:21.251610 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:21.252401 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:21.252550 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:21.253880 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:21.253888 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:21.253906 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:21.254609 (+ 703us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:21.254688 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:21.254713 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:21.254813 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:21.255192 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 02:09:21.255558 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":388,"threads_started":1}
I20260928 02:09:21.257459 20657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18: Bootstrap starting.
I20260928 02:09:21.260893 20657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:21.261844 20657 log.cc:824] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:21.265134 20657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18: No bootstrap required, opened a new log
I20260928 02:09:21.275370 20657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67aa46ddcfd749a1b89a54c38f033a18" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37105 } }
I20260928 02:09:21.275820 20657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:21.275955 20657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67aa46ddcfd749a1b89a54c38f033a18, State: Initialized, Role: FOLLOWER
I20260928 02:09:21.276605 20657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67aa46ddcfd749a1b89a54c38f033a18" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37105 } }
I20260928 02:09:21.276820 20657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:09:21.276930 20657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:09:21.277077 20657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:21.279296 20657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67aa46ddcfd749a1b89a54c38f033a18" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37105 } }
I20260928 02:09:21.279778 20657 leader_election.cc:302] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67aa46ddcfd749a1b89a54c38f033a18; no voters:
I20260928 02:09:21.280347 20657 leader_election.cc:288] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:09:21.280539 20663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:21.281298 20663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18 [term 1 LEADER]: Becoming Leader. State: Replica: 67aa46ddcfd749a1b89a54c38f033a18, State: Running, Role: LEADER
I20260928 02:09:21.281816 20663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67aa46ddcfd749a1b89a54c38f033a18" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37105 } }
I20260928 02:09:21.282686 20657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:09:21.285192 20665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67aa46ddcfd749a1b89a54c38f033a18. Latest consensus state: current_term: 1 leader_uuid: "67aa46ddcfd749a1b89a54c38f033a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67aa46ddcfd749a1b89a54c38f033a18" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37105 } } }
I20260928 02:09:21.285682 20665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:21.285168 20664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67aa46ddcfd749a1b89a54c38f033a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67aa46ddcfd749a1b89a54c38f033a18" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 37105 } } }
I20260928 02:09:21.285921 20664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:21.287357 20668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:09:21.292450 20668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:09:21.305730 20668 catalog_manager.cc:1347] Generated new cluster ID: 2603f5c270f5479ca18adeff24b74829
I20260928 02:09:21.305917 20668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:09:21.336608 20668 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:09:21.338003 20668 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:09:21.347235 20668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18: Generated new TSK 0
I20260928 02:09:21.348268 20668 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:09:21.361222 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:37105
--builtin_ntp_servers=127.1.182.148:36629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:09:21.724355 20682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:21.724641 20682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:21.724807 20682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:21.732230 20682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:21.732336 20682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:21.732527 20682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:09:21.741926 20682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:36629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:37105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:21.743361 20682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:21.745164 20682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:21.758694 20687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:21.758725 20690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:21.758793 20688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:21.759732 20682 server_base.cc:1034] running on GCE node
I20260928 02:09:21.760334 20682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:21.761337 20682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:21.762576 20682 hybrid_clock.cc:648] HybridClock initialized: now 1790561361762521 us; error 43 us; skew 500 ppm
I20260928 02:09:21.762902 20682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:09:21.765353 20682 webserver.cc:460] Webserver started at http://127.1.182.129:43151/ using document root <none> and password file <none>
I20260928 02:09:21.766073 20682 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:21.766204 20682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:21.766551 20682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:21.769490 20682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "1cf08eb18d9d442cb2b5837fd844fd16"
format_stamp: "Formatted at 2026-09-28 02:09:21 on dist-test-slave-23bp"
server_key: "3e0c3a8096c90e6ecba5298ab44bf14d"
server_key_iv: "66ede8107e8aef3495ce56e76516af46"
server_key_version: "encryptionkey@0"
I20260928 02:09:21.770270 20682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "1cf08eb18d9d442cb2b5837fd844fd16"
format_stamp: "Formatted at 2026-09-28 02:09:21 on dist-test-slave-23bp"
server_key: "3e0c3a8096c90e6ecba5298ab44bf14d"
server_key_iv: "66ede8107e8aef3495ce56e76516af46"
server_key_version: "encryptionkey@0"
I20260928 02:09:21.775547 20682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 02:09:21.779299 20696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:21.780680 20682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:09:21.780879 20682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "1cf08eb18d9d442cb2b5837fd844fd16"
format_stamp: "Formatted at 2026-09-28 02:09:21 on dist-test-slave-23bp"
server_key: "3e0c3a8096c90e6ecba5298ab44bf14d"
server_key_iv: "66ede8107e8aef3495ce56e76516af46"
server_key_version: "encryptionkey@0"
I20260928 02:09:21.781147 20682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:21.799723 20682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:21.800673 20682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:21.801097 20682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:21.802126 20682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:09:21.803901 20682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:09:21.804029 20682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:21.804162 20682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:09:21.804244 20682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:21.838068 20682 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:44273
I20260928 02:09:21.838120 20809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:44273 every 8 connection(s)
I20260928 02:09:21.839545 20682 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:09:21.841491 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 20682
I20260928 02:09:21.841674 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:09:21.842128 1754 external_mini_cluster.cc:1428] Setting key 142610aabce32444e18f03a09e61db67
I20260928 02:09:21.845291 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:37105
--builtin_ntp_servers=127.1.182.148:36629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:09:21.853951 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:21.842518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:41593 (local address 127.1.182.190:37105)
0928 02:09:21.843004 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:21.843011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:21.844540 (+ 1529us) server_negotiation.cc:407] Connection header received
0928 02:09:21.846492 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:21.846501 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:21.846623 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:21.846742 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:21.848093 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:21.849225 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:21.850762 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:21.851548 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:21.851690 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:21.853076 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:21.853084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:21.853092 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:21.853120 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:21.853174 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:21.853195 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:21.853301 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:21.853603 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 02:09:21.853687 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":242,"threads_started":1}
I20260928 02:09:21.855530 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:21.843244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:37105 (local address 127.1.182.129:41593)
0928 02:09:21.844350 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:21.844398 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:21.846188 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:21.847020 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:21.847030 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:21.847203 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:09:21.847927 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:21.847944 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:21.849368 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:21.849372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:21.850612 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:21.850624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:21.851774 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:21.851779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:21.851986 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:21.852744 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:09:21.852746 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:09:21.852787 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:09:21.852871 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:09:21.853316 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:09:21.853331 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:09:21.853341 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:09:21.853547 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 02:09:21.854008 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":297,"threads_started":1}
I20260928 02:09:21.857146 20810 heartbeater.cc:347] Connected to a master server at 127.1.182.190:37105
I20260928 02:09:21.857486 20810 heartbeater.cc:464] Registering TS with master...
I20260928 02:09:21.858312 20810 heartbeater.cc:511] Master 127.1.182.190:37105 requested a full tablet report, sending...
I20260928 02:09:21.860607 20622 ts_manager.cc:194] Registered new tserver with Master: 1cf08eb18d9d442cb2b5837fd844fd16 (127.1.182.129:44273)
I20260928 02:09:21.862154 20622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.182.129:41593
W20260928 02:09:22.181748 20814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:22.182082 20814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:22.182256 20814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:22.189797 20814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:22.189946 20814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:22.190136 20814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:09:22.199426 20814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:36629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:37105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:22.200796 20814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:22.202644 20814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:22.215830 20819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:22.215914 20823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:22.215879 20821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:22.216766 20814 server_base.cc:1034] running on GCE node
I20260928 02:09:22.217386 20814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:22.218375 20814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:22.219615 20814 hybrid_clock.cc:648] HybridClock initialized: now 1790561362219553 us; error 59 us; skew 500 ppm
I20260928 02:09:22.219949 20814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:09:22.222177 20814 webserver.cc:460] Webserver started at http://127.1.182.130:43303/ using document root <none> and password file <none>
I20260928 02:09:22.222882 20814 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:22.223019 20814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:22.223395 20814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:22.226275 20814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "44eab709a45e415ca3c6210ef66342d4"
format_stamp: "Formatted at 2026-09-28 02:09:22 on dist-test-slave-23bp"
server_key: "3afda91efbae2811dd05dc7e0f7d3d84"
server_key_iv: "bb178350d8d955331a56b1a9704c5026"
server_key_version: "encryptionkey@0"
I20260928 02:09:22.227159 20814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "44eab709a45e415ca3c6210ef66342d4"
format_stamp: "Formatted at 2026-09-28 02:09:22 on dist-test-slave-23bp"
server_key: "3afda91efbae2811dd05dc7e0f7d3d84"
server_key_iv: "bb178350d8d955331a56b1a9704c5026"
server_key_version: "encryptionkey@0"
I20260928 02:09:22.232780 20814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:09:22.236280 20829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:22.238042 20814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:09:22.238229 20814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "44eab709a45e415ca3c6210ef66342d4"
format_stamp: "Formatted at 2026-09-28 02:09:22 on dist-test-slave-23bp"
server_key: "3afda91efbae2811dd05dc7e0f7d3d84"
server_key_iv: "bb178350d8d955331a56b1a9704c5026"
server_key_version: "encryptionkey@0"
I20260928 02:09:22.238474 20814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:22.257531 20814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:22.258565 20814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:22.258926 20814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:22.260152 20814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:09:22.261873 20814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:09:22.261961 20814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:22.262037 20814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:09:22.262095 20814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:22.296070 20814 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:39259
I20260928 02:09:22.296128 20941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:39259 every 8 connection(s)
I20260928 02:09:22.297442 20814 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:09:22.303524 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 20814
I20260928 02:09:22.303718 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:09:22.304316 1754 external_mini_cluster.cc:1428] Setting key 10d78334d184023bf72ff654255717ae
I20260928 02:09:22.307461 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:37105
--builtin_ntp_servers=127.1.182.148:36629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:09:22.312493 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:22.300364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:50419 (local address 127.1.182.190:37105)
0928 02:09:22.300586 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:22.300593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:22.302137 (+ 1544us) server_negotiation.cc:407] Connection header received
0928 02:09:22.304603 (+ 2466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:22.304607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:22.304681 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:22.304770 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:22.306053 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.306975 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.309565 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.309957 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.310055 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:22.311756 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:22.311761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:22.311765 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:22.311783 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:22.311814 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:22.311827 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:22.311897 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:22.312187 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 02:09:22.312268 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:09:22.314025 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:22.301013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:37105 (local address 127.1.182.130:50419)
0928 02:09:22.301949 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:22.301992 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:22.304248 (+ 2256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:22.305057 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:22.305066 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:22.305202 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:09:22.305881 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:22.305898 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.307168 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.307176 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:22.309394 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:22.309411 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.310204 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.310214 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:22.310544 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:22.311385 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:09:22.311390 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:09:22.311451 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:09:22.311556 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:09:22.311926 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:09:22.311939 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:09:22.311951 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:09:22.312155 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 02:09:22.312578 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":294,"threads_started":1}
I20260928 02:09:22.315553 20942 heartbeater.cc:347] Connected to a master server at 127.1.182.190:37105
I20260928 02:09:22.315886 20942 heartbeater.cc:464] Registering TS with master...
I20260928 02:09:22.316825 20942 heartbeater.cc:511] Master 127.1.182.190:37105 requested a full tablet report, sending...
I20260928 02:09:22.318420 20622 ts_manager.cc:194] Registered new tserver with Master: 44eab709a45e415ca3c6210ef66342d4 (127.1.182.130:39259)
I20260928 02:09:22.319325 20622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.182.130:50419
W20260928 02:09:22.648794 20945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:22.649086 20945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:22.649240 20945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:22.656466 20945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:22.656570 20945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:22.656769 20945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:09:22.665899 20945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:36629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:37105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:22.667274 20945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:22.669102 20945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:22.681962 20953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:22.682008 20950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:22.682173 20951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:22.682938 20945 server_base.cc:1034] running on GCE node
I20260928 02:09:22.683573 20945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:22.684607 20945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:22.685854 20945 hybrid_clock.cc:648] HybridClock initialized: now 1790561362685780 us; error 60 us; skew 500 ppm
I20260928 02:09:22.686179 20945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:09:22.688443 20945 webserver.cc:460] Webserver started at http://127.1.182.131:43255/ using document root <none> and password file <none>
I20260928 02:09:22.689142 20945 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:22.689275 20945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:22.689594 20945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:22.692543 20945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "db646e31e999434bad228c04af3e3a18"
format_stamp: "Formatted at 2026-09-28 02:09:22 on dist-test-slave-23bp"
server_key: "6d68dc34f402f47b4fa336e923ff87be"
server_key_iv: "7203848b9a544026b288a5fd3d12dfd6"
server_key_version: "encryptionkey@0"
I20260928 02:09:22.693342 20945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "db646e31e999434bad228c04af3e3a18"
format_stamp: "Formatted at 2026-09-28 02:09:22 on dist-test-slave-23bp"
server_key: "6d68dc34f402f47b4fa336e923ff87be"
server_key_iv: "7203848b9a544026b288a5fd3d12dfd6"
server_key_version: "encryptionkey@0"
I20260928 02:09:22.698338 20945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:09:22.702014 20959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:22.703459 20945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 02:09:22.703630 20945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "db646e31e999434bad228c04af3e3a18"
format_stamp: "Formatted at 2026-09-28 02:09:22 on dist-test-slave-23bp"
server_key: "6d68dc34f402f47b4fa336e923ff87be"
server_key_iv: "7203848b9a544026b288a5fd3d12dfd6"
server_key_version: "encryptionkey@0"
I20260928 02:09:22.703846 20945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:22.728184 20945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:22.729077 20945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:22.729399 20945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:22.730397 20945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:09:22.731885 20945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:09:22.732028 20945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:22.732185 20945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:09:22.732282 20945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:22.765898 20945 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:46051
I20260928 02:09:22.765986 21072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:46051 every 8 connection(s)
I20260928 02:09:22.767380 20945 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:09:22.775445 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 20945
I20260928 02:09:22.775640 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:09:22.776156 1754 external_mini_cluster.cc:1428] Setting key 4742f61ede28de5165891cc309d5ad94
I20260928 02:09:22.780686 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:22.769981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:50441 (local address 127.1.182.190:37105)
0928 02:09:22.770190 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:22.770195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:22.771686 (+ 1491us) server_negotiation.cc:407] Connection header received
0928 02:09:22.773500 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:22.773504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:22.773578 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:22.773668 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:22.774701 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.775768 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.777907 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.778415 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.778511 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:22.780013 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:22.780017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:22.780021 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:22.780038 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:22.780072 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:22.780084 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:22.780155 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:22.780353 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 02:09:22.780430 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:09:22.781713 21075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:22.770559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:37105 (local address 127.1.182.131:50441)
0928 02:09:22.771519 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:22.771557 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:22.773250 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:22.773873 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:22.773881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:22.773975 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:09:22.774559 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:22.774575 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.775937 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.775941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:22.777722 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:22.777737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.778662 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.778669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:22.778976 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:22.779710 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:09:22.779713 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:09:22.779755 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:09:22.779837 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:09:22.780169 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:09:22.780176 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:09:22.780182 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:09:22.780324 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 02:09:22.780637 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":264,"threads_started":1}
I20260928 02:09:22.783206 21073 heartbeater.cc:347] Connected to a master server at 127.1.182.190:37105
I20260928 02:09:22.783516 21073 heartbeater.cc:464] Registering TS with master...
I20260928 02:09:22.784304 21073 heartbeater.cc:511] Master 127.1.182.190:37105 requested a full tablet report, sending...
I20260928 02:09:22.785784 20622 ts_manager.cc:194] Registered new tserver with Master: db646e31e999434bad228c04af3e3a18 (127.1.182.131:46051)
I20260928 02:09:22.786736 20622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.182.131:50441
I20260928 02:09:22.791404 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:09:22.801491 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:22.795638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46072 (local address 127.1.182.190:37105)
0928 02:09:22.795862 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:22.795871 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:22.796208 (+ 337us) server_negotiation.cc:407] Connection header received
0928 02:09:22.796344 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:22.796353 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:22.796420 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:22.796507 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:22.796975 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.797667 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.799049 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.799537 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.799639 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:22.800694 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:22.800698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:22.800701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:22.800719 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:22.800756 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:22.800768 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:22.800934 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:22.801132 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 02:09:22.801191 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:09:22.805430 20622 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:09:22.812614 20622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:09:22.853493 21085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:22.825403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:39259 (local address 127.0.0.1:47326)
0928 02:09:22.841311 (+ 15908us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:22.841352 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:22.841543 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:22.842867 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:22.842874 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:22.842908 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:22.843356 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:22.843377 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.846107 (+ 2730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.846117 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:22.849638 (+ 3521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:22.849664 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.852407 (+ 2743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.852415 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:22.852680 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:22.852745 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 02:09:22.852852 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:09:22.853043 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15630,"thread_start_us":15434,"threads_started":1}
I20260928 02:09:22.854555 21086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:22.825403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47326 (local address 127.1.182.130:39259)
0928 02:09:22.842076 (+ 16673us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:22.842089 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:22.842130 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:09:22.842256 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:22.842267 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:22.842620 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:22.842842 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:22.843724 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.845895 (+ 2171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.850016 (+ 4121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.852150 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.853773 (+ 1623us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:22.854090 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:22.854154 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 02:09:22.854293 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16423,"thread_start_us":17655,"threads_started":1}
I20260928 02:09:22.858584 20877 tablet_service.cc:1467] Processing CreateTablet for tablet 31c2a4e5e1ae4c2d95b5f0a314abd895 (DEFAULT_TABLE table=test-table [id=ca9921f3d5ec446e865860301a5b7205]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:09:22.860175 20877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c2a4e5e1ae4c2d95b5f0a314abd895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:22.860224 21087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:22.825491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:46051 (local address 127.0.0.1:41740)
0928 02:09:22.842693 (+ 17202us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:22.842714 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:22.842830 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:22.853409 (+ 10579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:22.853415 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:22.853450 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:22.853878 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:22.853890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.855959 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.855964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:22.857453 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:22.857466 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.859204 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.859216 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:22.859587 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:22.859611 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:09:22.859711 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 02:09:22.859844 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15800,"mutex_wait_us":101,"thread_start_us":17451,"threads_started":1}
I20260928 02:09:22.860320 21088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:22.826136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41740 (local address 127.1.182.131:46051)
0928 02:09:22.852591 (+ 26455us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:22.852605 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:22.852640 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:09:22.852745 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:22.852753 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:22.853120 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:22.853316 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:22.854120 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.855799 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.857643 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.858974 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.859156 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:22.859419 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:22.859766 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 02:09:22.859917 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26283,"thread_start_us":26131,"threads_started":1}
I20260928 02:09:22.863055 21089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:22.827133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:44273 (local address 127.0.0.1:40896)
0928 02:09:22.852248 (+ 25115us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:22.852274 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:22.852428 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:22.853088 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:22.853092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:22.853119 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:22.853540 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:22.853555 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.857824 (+ 4269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.857832 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:22.859364 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:22.859378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.862263 (+ 2885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.862270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:22.862476 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:22.862529 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 02:09:22.862634 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:09:22.862764 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24944,"thread_start_us":24731,"threads_started":1}
I20260928 02:09:22.863288 21090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:22.827208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40896 (local address 127.1.182.129:44273)
0928 02:09:22.851667 (+ 24459us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:22.851680 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:22.852550 (+ 870us) server_negotiation.cc:407] Connection header received
0928 02:09:22.852660 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:22.852667 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:22.852886 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:22.853114 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:22.853978 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.857698 (+ 3720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.859556 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.862038 (+ 2482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.862203 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:22.862477 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:22.862696 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 02:09:22.862866 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24219,"thread_start_us":24095,"threads_started":1}
I20260928 02:09:22.864421 21008 tablet_service.cc:1467] Processing CreateTablet for tablet 31c2a4e5e1ae4c2d95b5f0a314abd895 (DEFAULT_TABLE table=test-table [id=ca9921f3d5ec446e865860301a5b7205]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:09:22.865978 21008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c2a4e5e1ae4c2d95b5f0a314abd895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:22.867331 20810 heartbeater.cc:503] Master 127.1.182.190:37105 was elected leader, sending a full tablet report...
I20260928 02:09:22.867465 20745 tablet_service.cc:1467] Processing CreateTablet for tablet 31c2a4e5e1ae4c2d95b5f0a314abd895 (DEFAULT_TABLE table=test-table [id=ca9921f3d5ec446e865860301a5b7205]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:09:22.869043 20745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c2a4e5e1ae4c2d95b5f0a314abd895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:22.873245 21091 tablet_bootstrap.cc:492] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 44eab709a45e415ca3c6210ef66342d4: Bootstrap starting.
I20260928 02:09:22.877938 21091 tablet_bootstrap.cc:654] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 44eab709a45e415ca3c6210ef66342d4: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:22.879019 21091 log.cc:824] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 44eab709a45e415ca3c6210ef66342d4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:22.881048 21093 tablet_bootstrap.cc:492] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18: Bootstrap starting.
I20260928 02:09:22.882586 21091 tablet_bootstrap.cc:492] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 44eab709a45e415ca3c6210ef66342d4: No bootstrap required, opened a new log
I20260928 02:09:22.882879 21091 ts_tablet_manager.cc:1397] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 44eab709a45e415ca3c6210ef66342d4: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260928 02:09:22.883698 21094 tablet_bootstrap.cc:492] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 1cf08eb18d9d442cb2b5837fd844fd16: Bootstrap starting.
I20260928 02:09:22.884975 21093 tablet_bootstrap.cc:654] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:22.886202 21093 log.cc:824] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:22.886821 21094 tablet_bootstrap.cc:654] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 1cf08eb18d9d442cb2b5837fd844fd16: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:22.888051 21094 log.cc:824] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 1cf08eb18d9d442cb2b5837fd844fd16: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:22.889535 21093 tablet_bootstrap.cc:492] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18: No bootstrap required, opened a new log
I20260928 02:09:22.889812 21093 ts_tablet_manager.cc:1397] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260928 02:09:22.893520 21094 tablet_bootstrap.cc:492] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 1cf08eb18d9d442cb2b5837fd844fd16: No bootstrap required, opened a new log
I20260928 02:09:22.893787 21094 ts_tablet_manager.cc:1397] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 1cf08eb18d9d442cb2b5837fd844fd16: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260928 02:09:22.894785 21091 raft_consensus.cc:348] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 44eab709a45e415ca3c6210ef66342d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44eab709a45e415ca3c6210ef66342d4" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39259 } } peers { permanent_uuid: "db646e31e999434bad228c04af3e3a18" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46051 } } peers { permanent_uuid: "1cf08eb18d9d442cb2b5837fd844fd16" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44273 } }
I20260928 02:09:22.895325 21091 raft_consensus.cc:374] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 44eab709a45e415ca3c6210ef66342d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:22.895432 21091 raft_consensus.cc:729] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 44eab709a45e415ca3c6210ef66342d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44eab709a45e415ca3c6210ef66342d4, State: Initialized, Role: FOLLOWER
I20260928 02:09:22.896070 21091 consensus_queue.cc:260] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 44eab709a45e415ca3c6210ef66342d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44eab709a45e415ca3c6210ef66342d4" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39259 } } peers { permanent_uuid: "db646e31e999434bad228c04af3e3a18" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46051 } } peers { permanent_uuid: "1cf08eb18d9d442cb2b5837fd844fd16" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44273 } }
I20260928 02:09:22.897198 20942 heartbeater.cc:503] Master 127.1.182.190:37105 was elected leader, sending a full tablet report...
I20260928 02:09:22.897833 21091 ts_tablet_manager.cc:1428] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 44eab709a45e415ca3c6210ef66342d4: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260928 02:09:22.900197 21093 raft_consensus.cc:348] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44eab709a45e415ca3c6210ef66342d4" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39259 } } peers { permanent_uuid: "db646e31e999434bad228c04af3e3a18" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46051 } } peers { permanent_uuid: "1cf08eb18d9d442cb2b5837fd844fd16" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44273 } }
I20260928 02:09:22.900660 21093 raft_consensus.cc:374] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:22.900755 21093 raft_consensus.cc:729] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db646e31e999434bad228c04af3e3a18, State: Initialized, Role: FOLLOWER
I20260928 02:09:22.901386 21093 consensus_queue.cc:260] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44eab709a45e415ca3c6210ef66342d4" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39259 } } peers { permanent_uuid: "db646e31e999434bad228c04af3e3a18" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46051 } } peers { permanent_uuid: "1cf08eb18d9d442cb2b5837fd844fd16" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44273 } }
I20260928 02:09:22.902576 21073 heartbeater.cc:503] Master 127.1.182.190:37105 was elected leader, sending a full tablet report...
I20260928 02:09:22.903448 21093 ts_tablet_manager.cc:1428] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260928 02:09:22.904628 21094 raft_consensus.cc:348] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 1cf08eb18d9d442cb2b5837fd844fd16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44eab709a45e415ca3c6210ef66342d4" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39259 } } peers { permanent_uuid: "db646e31e999434bad228c04af3e3a18" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46051 } } peers { permanent_uuid: "1cf08eb18d9d442cb2b5837fd844fd16" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44273 } }
I20260928 02:09:22.905067 21094 raft_consensus.cc:374] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 1cf08eb18d9d442cb2b5837fd844fd16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:22.905191 21094 raft_consensus.cc:729] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 1cf08eb18d9d442cb2b5837fd844fd16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf08eb18d9d442cb2b5837fd844fd16, State: Initialized, Role: FOLLOWER
I20260928 02:09:22.905799 21094 consensus_queue.cc:260] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 1cf08eb18d9d442cb2b5837fd844fd16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44eab709a45e415ca3c6210ef66342d4" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39259 } } peers { permanent_uuid: "db646e31e999434bad228c04af3e3a18" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46051 } } peers { permanent_uuid: "1cf08eb18d9d442cb2b5837fd844fd16" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44273 } }
I20260928 02:09:22.907459 21094 ts_tablet_manager.cc:1428] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 1cf08eb18d9d442cb2b5837fd844fd16: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260928 02:09:22.914919 21098 raft_consensus.cc:482] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:09:22.915228 21098 raft_consensus.cc:504] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44eab709a45e415ca3c6210ef66342d4" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39259 } } peers { permanent_uuid: "db646e31e999434bad228c04af3e3a18" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46051 } } peers { permanent_uuid: "1cf08eb18d9d442cb2b5837fd844fd16" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44273 } }
I20260928 02:09:22.916821 21098 leader_election.cc:288] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44eab709a45e415ca3c6210ef66342d4 (127.1.182.130:39259), 1cf08eb18d9d442cb2b5837fd844fd16 (127.1.182.129:44273)
I20260928 02:09:22.922441 21075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:22.917071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:44273 (local address 127.1.182.131:37341)
0928 02:09:22.917362 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:22.917382 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:22.917526 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:22.917922 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:22.917950 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:22.917986 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:22.918229 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:22.918238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.919256 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.919263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:22.920637 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:22.920649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.921724 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.921729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:22.921904 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:22.921928 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:09:22.922027 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 02:09:22.922164 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260928 02:09:22.922441 21090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:22.917219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:37341 (local address 127.1.182.129:44273)
0928 02:09:22.917390 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:22.917396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:22.917497 (+ 101us) server_negotiation.cc:407] Connection header received
0928 02:09:22.917695 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:22.917700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:22.917768 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:22.917867 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:22.918379 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.919128 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.920787 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.921565 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.921730 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:22.921800 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:22.922071 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 02:09:22.922191 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:09:22.923339 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:22.917262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:39259 (local address 127.1.182.131:60287)
0928 02:09:22.918171 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:22.918191 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:22.918339 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:22.918682 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:22.918686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:22.918708 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:22.919110 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:22.919124 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.920115 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.920121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:22.921487 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:22.921498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.922622 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.922627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:22.922812 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:22.922850 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:09:22.922950 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 02:09:22.923096 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":426,"threads_started":1}
I20260928 02:09:22.923445 21086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:22.917250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:60287 (local address 127.1.182.130:39259)
0928 02:09:22.917479 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:22.917487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:22.918278 (+ 791us) server_negotiation.cc:407] Connection header received
0928 02:09:22.918484 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:22.918489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:22.918563 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:22.918669 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:22.919297 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.919959 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.921677 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.922443 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.922630 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:22.922697 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:22.922999 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 02:09:22.923171 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 02:09:22.923689 20765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c2a4e5e1ae4c2d95b5f0a314abd895" candidate_uuid: "db646e31e999434bad228c04af3e3a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cf08eb18d9d442cb2b5837fd844fd16" is_pre_election: true
I20260928 02:09:22.924302 20765 raft_consensus.cc:2454] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 1cf08eb18d9d442cb2b5837fd844fd16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db646e31e999434bad228c04af3e3a18 in term 0.
I20260928 02:09:22.924412 20897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c2a4e5e1ae4c2d95b5f0a314abd895" candidate_uuid: "db646e31e999434bad228c04af3e3a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44eab709a45e415ca3c6210ef66342d4" is_pre_election: true
I20260928 02:09:22.924998 20897 raft_consensus.cc:2454] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 44eab709a45e415ca3c6210ef66342d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db646e31e999434bad228c04af3e3a18 in term 0.
I20260928 02:09:22.924993 20964 leader_election.cc:302] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cf08eb18d9d442cb2b5837fd844fd16, db646e31e999434bad228c04af3e3a18; no voters:
I20260928 02:09:22.925383 21098 raft_consensus.cc:2781] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:09:22.925544 21098 raft_consensus.cc:482] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:09:22.925650 21098 raft_consensus.cc:3037] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:22.928007 21098 raft_consensus.cc:504] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44eab709a45e415ca3c6210ef66342d4" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39259 } } peers { permanent_uuid: "db646e31e999434bad228c04af3e3a18" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46051 } } peers { permanent_uuid: "1cf08eb18d9d442cb2b5837fd844fd16" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44273 } }
I20260928 02:09:22.928555 21098 leader_election.cc:288] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 [CANDIDATE]: Term 1 election: Requested vote from peers 44eab709a45e415ca3c6210ef66342d4 (127.1.182.130:39259), 1cf08eb18d9d442cb2b5837fd844fd16 (127.1.182.129:44273)
I20260928 02:09:22.929030 20897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c2a4e5e1ae4c2d95b5f0a314abd895" candidate_uuid: "db646e31e999434bad228c04af3e3a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44eab709a45e415ca3c6210ef66342d4"
I20260928 02:09:22.929030 20765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c2a4e5e1ae4c2d95b5f0a314abd895" candidate_uuid: "db646e31e999434bad228c04af3e3a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cf08eb18d9d442cb2b5837fd844fd16"
I20260928 02:09:22.929241 20765 raft_consensus.cc:3037] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 1cf08eb18d9d442cb2b5837fd844fd16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:22.929241 20897 raft_consensus.cc:3037] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 44eab709a45e415ca3c6210ef66342d4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:22.932598 20897 raft_consensus.cc:2454] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 44eab709a45e415ca3c6210ef66342d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db646e31e999434bad228c04af3e3a18 in term 1.
I20260928 02:09:22.932605 20765 raft_consensus.cc:2454] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 1cf08eb18d9d442cb2b5837fd844fd16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db646e31e999434bad228c04af3e3a18 in term 1.
I20260928 02:09:22.933044 20964 leader_election.cc:302] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cf08eb18d9d442cb2b5837fd844fd16, db646e31e999434bad228c04af3e3a18; no voters:
I20260928 02:09:22.933359 21098 raft_consensus.cc:2781] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:22.933825 21098 raft_consensus.cc:686] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 [term 1 LEADER]: Becoming Leader. State: Replica: db646e31e999434bad228c04af3e3a18, State: Running, Role: LEADER
I20260928 02:09:22.934345 21098 consensus_queue.cc:237] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44eab709a45e415ca3c6210ef66342d4" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39259 } } peers { permanent_uuid: "db646e31e999434bad228c04af3e3a18" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46051 } } peers { permanent_uuid: "1cf08eb18d9d442cb2b5837fd844fd16" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44273 } }
I20260928 02:09:22.939240 20622 catalog_manager.cc:5495] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 reported cstate change: term changed from 0 to 1, leader changed from <none> to db646e31e999434bad228c04af3e3a18 (127.1.182.131). New cstate: current_term: 1 leader_uuid: "db646e31e999434bad228c04af3e3a18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44eab709a45e415ca3c6210ef66342d4" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db646e31e999434bad228c04af3e3a18" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 46051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cf08eb18d9d442cb2b5837fd844fd16" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44273 } health_report { overall_health: UNKNOWN } } }
I20260928 02:09:22.962519 21088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:22.956755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.1.182.131:46051)
0928 02:09:22.956967 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:22.956973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:22.957087 (+ 114us) server_negotiation.cc:407] Connection header received
0928 02:09:22.957340 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:22.957344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:22.957430 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:22.957588 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:09:22.957993 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.958722 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.960267 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:22.960690 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:22.960826 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:22.961433 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:09:22.961929 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:09:22.962131 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:22.962242 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 02:09:22.962287 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:09:22.971297 20897 raft_consensus.cc:1261] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 44eab709a45e415ca3c6210ef66342d4 [term 1 FOLLOWER]: Refusing update from remote peer db646e31e999434bad228c04af3e3a18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:09:22.971292 20765 raft_consensus.cc:1261] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 1cf08eb18d9d442cb2b5837fd844fd16 [term 1 FOLLOWER]: Refusing update from remote peer db646e31e999434bad228c04af3e3a18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:09:22.972501 21101 consensus_queue.cc:1035] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44eab709a45e415ca3c6210ef66342d4" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 39259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:09:22.972886 21098 consensus_queue.cc:1035] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cf08eb18d9d442cb2b5837fd844fd16" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 44273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:09:22.989063 21106 mvcc.cc:205] Tried to move back new op lower bound from 7334139342719070208 to 7334139342583046144. Current Snapshot: MvccSnapshot[applied={T|T < 7334139342719070208}]
I20260928 02:09:23.007542 20621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:09:23.007876 20621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:09:23.011667 20621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67aa46ddcfd749a1b89a54c38f033a18: Sending DeleteTablet for 3 replicas of tablet 31c2a4e5e1ae4c2d95b5f0a314abd895
I20260928 02:09:23.012794 20877 tablet_service.cc:1514] Processing DeleteTablet for tablet 31c2a4e5e1ae4c2d95b5f0a314abd895 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:09:23 UTC) from {username='slave'} at 127.0.0.1:47326
I20260928 02:09:23.012910 20745 tablet_service.cc:1514] Processing DeleteTablet for tablet 31c2a4e5e1ae4c2d95b5f0a314abd895 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:09:23 UTC) from {username='slave'} at 127.0.0.1:40896
I20260928 02:09:23.012914 21008 tablet_service.cc:1514] Processing DeleteTablet for tablet 31c2a4e5e1ae4c2d95b5f0a314abd895 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:09:23 UTC) from {username='slave'} at 127.0.0.1:41740
I20260928 02:09:23.015767 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 20682
I20260928 02:09:23.028780 21118 tablet_replica.cc:330] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18: stopping tablet replica
I20260928 02:09:23.031283 21118 raft_consensus.cc:2229] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:09:23.032218 21118 raft_consensus.cc:2258] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:09:23.035580 21116 tablet_replica.cc:330] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 44eab709a45e415ca3c6210ef66342d4: stopping tablet replica
I20260928 02:09:23.036072 21116 raft_consensus.cc:2229] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 44eab709a45e415ca3c6210ef66342d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:09:23.036655 21116 raft_consensus.cc:2258] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 44eab709a45e415ca3c6210ef66342d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:09:23.038522 21118 ts_tablet_manager.cc:1905] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:09:23.039870 21117 tablet_replica.cc:330] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 1cf08eb18d9d442cb2b5837fd844fd16: stopping tablet replica
I20260928 02:09:23.040365 21117 raft_consensus.cc:2229] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 1cf08eb18d9d442cb2b5837fd844fd16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:09:23.040994 21117 raft_consensus.cc:2258] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 1cf08eb18d9d442cb2b5837fd844fd16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:09:23.042012 21090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:23.017534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40906 (local address 127.1.182.129:44273)
0928 02:09:23.017742 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:23.017752 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:23.023470 (+ 5718us) server_negotiation.cc:407] Connection header received
0928 02:09:23.024459 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:23.024467 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:23.024583 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:23.024672 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:23.025920 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:23.027317 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:23.032496 (+ 5179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:23.033274 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:23.033383 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:23.040108 (+ 6725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:23.040120 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:23.040136 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:23.040925 (+ 789us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:23.040995 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:23.041020 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:23.041139 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:23.041439 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 02:09:23.041575 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 02:09:23.043322 20805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:23.043424 21116 ts_tablet_manager.cc:1905] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 44eab709a45e415ca3c6210ef66342d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:09:23.047796 21118 ts_tablet_manager.cc:1918] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:09:23.047981 21118 log.cc:1196] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/31c2a4e5e1ae4c2d95b5f0a314abd895
I20260928 02:09:23.048826 21118 ts_tablet_manager.cc:1939] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P db646e31e999434bad228c04af3e3a18: Deleting consensus metadata
I20260928 02:09:23.050943 20608 catalog_manager.cc:4841] TS db646e31e999434bad228c04af3e3a18 (127.1.182.131:46051): tablet 31c2a4e5e1ae4c2d95b5f0a314abd895 (table test-table [id=ca9921f3d5ec446e865860301a5b7205]) successfully deleted
I20260928 02:09:23.055542 21116 ts_tablet_manager.cc:1918] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 44eab709a45e415ca3c6210ef66342d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:09:23.055713 21116 log.cc:1196] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 44eab709a45e415ca3c6210ef66342d4: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/31c2a4e5e1ae4c2d95b5f0a314abd895
I20260928 02:09:23.056286 21116 ts_tablet_manager.cc:1939] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 44eab709a45e415ca3c6210ef66342d4: Deleting consensus metadata
I20260928 02:09:23.058215 20607 catalog_manager.cc:4841] TS 44eab709a45e415ca3c6210ef66342d4 (127.1.182.130:39259): tablet 31c2a4e5e1ae4c2d95b5f0a314abd895 (table test-table [id=ca9921f3d5ec446e865860301a5b7205]) successfully deleted
I20260928 02:09:23.043361 21117 ts_tablet_manager.cc:1905] T 31c2a4e5e1ae4c2d95b5f0a314abd895 P 1cf08eb18d9d442cb2b5837fd844fd16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:09:23.142799 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 20682
W20260928 02:09:23.161432 20609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.182.129:44273 (error 108)
I20260928 02:09:23.161693 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 20814
I20260928 02:09:23.163885 21089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:23.163406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:44273 (local address 127.0.0.1:40916)
0928 02:09:23.163601 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:23.163707 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:44273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 02:09:23.164886 20609 catalog_manager.cc:4584] TS 1cf08eb18d9d442cb2b5837fd844fd16 (127.1.182.129:44273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31c2a4e5e1ae4c2d95b5f0a314abd895: Network error: Client connection negotiation failed: client connection to 127.1.182.129:44273: connect: Connection refused (error 111)
I20260928 02:09:23.169441 21086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:23.162547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47340 (local address 127.1.182.130:39259)
0928 02:09:23.162738 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:23.162745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:23.162865 (+ 120us) server_negotiation.cc:407] Connection header received
0928 02:09:23.163005 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:23.163010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:23.163158 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:23.163272 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:23.163890 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:23.164681 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:23.166549 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:23.166943 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:23.167034 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:23.168193 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:23.168202 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:23.168217 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:23.168797 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:23.168842 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:23.168861 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:23.168947 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:23.169104 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 02:09:23.169190 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:09:23.170100 20937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:23.218858 21089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:23.218198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:44273 (local address 127.0.0.1:40918)
0928 02:09:23.218439 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:23.218664 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:44273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 02:09:23.220034 21089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:23.219637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:44273 (local address 127.0.0.1:40926)
0928 02:09:23.219797 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:23.219889 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:44273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 02:09:23.275978 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 20814
I20260928 02:09:23.280782 21089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:23.280137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:44273 (local address 127.0.0.1:40942)
0928 02:09:23.280454 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:23.280557 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:44273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260928 02:09:23.282159 21089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:23.281678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:44273 (local address 127.0.0.1:40958)
0928 02:09:23.281903 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:23.281994 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:44273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260928 02:09:23.300501 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 20945
I20260928 02:09:23.309520 21088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:23.301383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41760 (local address 127.1.182.131:46051)
0928 02:09:23.301596 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:23.301602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:23.301627 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:09:23.301764 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:23.301770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:23.301892 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:23.302013 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:23.302452 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:23.303919 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:23.305431 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:23.306210 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:23.306351 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:23.307716 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:23.307728 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:23.307749 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:23.308648 (+ 899us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:23.308712 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:23.308738 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:23.308864 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:23.308982 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 02:09:23.309082 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 02:09:23.310314 21068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:23.362937 21089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:23.362393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:44273 (local address 127.0.0.1:40972)
0928 02:09:23.362640 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:23.362751 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:44273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 02:09:23.364082 21089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:23.363648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:44273 (local address 127.0.0.1:40986)
0928 02:09:23.363864 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:23.363948 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:44273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 02:09:23.433863 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 20945
I20260928 02:09:23.456554 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 20591
I20260928 02:09:23.457628 20652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:23.569164 21089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:23.568296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:44273 (local address 127.0.0.1:40996)
0928 02:09:23.568688 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:23.568864 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:44273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260928 02:09:23.569561 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 20591
2026-09-28T02:09:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2851 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:09:23 dist-test-slave-23bp krb5kdc[21127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:09:23 dist-test-slave-23bp krb5kdc[21127](info): set up 2 sockets
Sep 28 02:09:23 dist-test-slave-23bp krb5kdc[21127](info): commencing operation
krb5kdc: starting...
W20260928 02:09:24.713845 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:09:24 dist-test-slave-23bp krb5kdc[21127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:09:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:09:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:09:24.974035 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:42835
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:42835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:25.310245 21140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:25.310547 21140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:25.310703 21140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:25.318413 21140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:09:25.318526 21140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:25.318624 21140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:25.318729 21140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:09:25.318811 21140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:09:25.329047 21140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:42835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:42835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:25.330474 21140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:25.332016 21140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:25.343945 21145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:25.343997 21148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:25.343904 21146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:25.344998 21140 server_base.cc:1034] running on GCE node
I20260928 02:09:25.346019 21140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:25.347733 21140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:25.349009 21140 hybrid_clock.cc:648] HybridClock initialized: now 1790561365348955 us; error 55 us; skew 500 ppm
I20260928 02:09:25.349442 21140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:25 dist-test-slave-23bp krb5kdc[21127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:25.358189 21140 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:09:25.359906 21140 webserver.cc:460] Webserver started at http://127.1.182.190:45475/ using document root <none> and password file <none>
I20260928 02:09:25.360630 21140 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:25.360780 21140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:25.361114 21140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:25.364219 21140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "e989430d06514bc7b22e67f7f0227cbe"
format_stamp: "Formatted at 2026-09-28 02:09:25 on dist-test-slave-23bp"
server_key: "9831e5a20d96ad5b2fefb9fa7b579fdb"
server_key_iv: "0cb3cb907f27f4d47032027402418394"
server_key_version: "encryptionkey@0"
I20260928 02:09:25.364996 21140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "e989430d06514bc7b22e67f7f0227cbe"
format_stamp: "Formatted at 2026-09-28 02:09:25 on dist-test-slave-23bp"
server_key: "9831e5a20d96ad5b2fefb9fa7b579fdb"
server_key_iv: "0cb3cb907f27f4d47032027402418394"
server_key_version: "encryptionkey@0"
I20260928 02:09:25.370348 21140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:09:25.373814 21155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:25.375166 21140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 02:09:25.375349 21140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "e989430d06514bc7b22e67f7f0227cbe"
format_stamp: "Formatted at 2026-09-28 02:09:25 on dist-test-slave-23bp"
server_key: "9831e5a20d96ad5b2fefb9fa7b579fdb"
server_key_iv: "0cb3cb907f27f4d47032027402418394"
server_key_version: "encryptionkey@0"
I20260928 02:09:25.375571 21140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:25.394436 21140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:25.399001 21140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:25.399317 21140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:25.418040 21140 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:42835
I20260928 02:09:25.418135 21206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:42835 every 8 connection(s)
I20260928 02:09:25.419507 21140 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:09:25.423643 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:25.428576 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 21140
I20260928 02:09:25.428766 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:09:25.429186 1754 external_mini_cluster.cc:1428] Setting key b21bcf8827bc877105c593d0517db5f1
I20260928 02:09:25.436081 21207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe: Bootstrap starting.
I20260928 02:09:25.439836 21207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:25.441045 21207 log.cc:824] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:25.444147 21207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe: No bootstrap required, opened a new log
Sep 28 02:09:25 dist-test-slave-23bp krb5kdc[21127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:09:25.453907 21207 raft_consensus.cc:348] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e989430d06514bc7b22e67f7f0227cbe" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 42835 } }
I20260928 02:09:25.454159 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:25.431014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48332 (local address 127.1.182.190:42835)
0928 02:09:25.431960 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:25.431978 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:25.432031 (+ 53us) server_negotiation.cc:407] Connection header received
0928 02:09:25.434586 (+ 2555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:25.434613 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:25.435133 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:25.435580 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:25.437096 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:25.439208 (+ 2112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:25.440535 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:25.441656 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:25.441836 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:25.446503 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:25.446511 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:25.446525 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:25.446570 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:25.449791 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:25.450762 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:25.450768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:25.450774 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:25.450937 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:25.451390 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:25.451393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:25.451395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:25.451956 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:25.452193 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:25.452693 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 02:09:25.453123 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":421,"threads_started":1}
I20260928 02:09:25.454329 21207 raft_consensus.cc:374] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:25.454442 21207 raft_consensus.cc:729] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e989430d06514bc7b22e67f7f0227cbe, State: Initialized, Role: FOLLOWER
I20260928 02:09:25.455010 21207 consensus_queue.cc:260] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e989430d06514bc7b22e67f7f0227cbe" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 42835 } }
I20260928 02:09:25.455344 21207 raft_consensus.cc:388] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:09:25.455500 21207 raft_consensus.cc:482] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:09:25.455686 21207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:25.458227 21207 raft_consensus.cc:504] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e989430d06514bc7b22e67f7f0227cbe" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 42835 } }
I20260928 02:09:25.458791 21207 leader_election.cc:302] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e989430d06514bc7b22e67f7f0227cbe; no voters:
I20260928 02:09:25.459452 21207 leader_election.cc:288] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:09:25.459604 21212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:25.461223 21212 raft_consensus.cc:686] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe [term 1 LEADER]: Becoming Leader. State: Replica: e989430d06514bc7b22e67f7f0227cbe, State: Running, Role: LEADER
I20260928 02:09:25.461781 21207 sys_catalog.cc:564] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:09:25.461781 21212 consensus_queue.cc:237] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e989430d06514bc7b22e67f7f0227cbe" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 42835 } }
I20260928 02:09:25.465987 21214 sys_catalog.cc:455] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader e989430d06514bc7b22e67f7f0227cbe. Latest consensus state: current_term: 1 leader_uuid: "e989430d06514bc7b22e67f7f0227cbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e989430d06514bc7b22e67f7f0227cbe" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 42835 } } }
I20260928 02:09:25.465956 21213 sys_catalog.cc:455] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e989430d06514bc7b22e67f7f0227cbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e989430d06514bc7b22e67f7f0227cbe" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 42835 } } }
I20260928 02:09:25.466441 21214 sys_catalog.cc:458] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:25.466441 21213 sys_catalog.cc:458] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:25.469415 21218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:09:25.474951 21218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:09:25.484282 21218 catalog_manager.cc:1347] Generated new cluster ID: d90ddb7d4fdc4c5aa6782bf77ed415b5
I20260928 02:09:25.484401 21218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:09:25.503820 21218 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:09:25.505821 21218 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:09:25.521477 21218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe: Generated new TSK 0
I20260928 02:09:25.522646 21218 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.182.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
Entry for principal kudu/127.1.182.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab.
I20260928 02:09:25.586635 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:42835
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:25.924252 21233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:25.924541 21233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:25.924664 21233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:25.932271 21233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:25.932390 21233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:25.932502 21233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:09:25.942737 21233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.129.keytab
--principal=kudu/127.1.182.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:42835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:25.944149 21233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:25.945454 21233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:25.959213 21239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:25.959300 21241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:25.959249 21238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:25.960081 21233 server_base.cc:1034] running on GCE node
I20260928 02:09:25.960685 21233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:25.961694 21233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:25.962900 21233 hybrid_clock.cc:648] HybridClock initialized: now 1790561365962832 us; error 60 us; skew 500 ppm
I20260928 02:09:25.963282 21233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:25 dist-test-slave-23bp krb5kdc[21127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:25.966542 21233 init.cc:377] Logged in from keytab as kudu/127.1.182.129@KRBTEST.COM (short username kudu)
I20260928 02:09:25.968242 21233 webserver.cc:460] Webserver started at http://127.1.182.129:34977/ using document root <none> and password file <none>
I20260928 02:09:25.969067 21233 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:25.969213 21233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:25.969565 21233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:25.972759 21233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "aec5de72bb9e4d37bf649c96a8a483df"
format_stamp: "Formatted at 2026-09-28 02:09:25 on dist-test-slave-23bp"
server_key: "31b426d16f71cfe9f91c3d27e022f0f4"
server_key_iv: "7b0e8364eb7ae0864c45165949c9365b"
server_key_version: "encryptionkey@0"
I20260928 02:09:25.973560 21233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "aec5de72bb9e4d37bf649c96a8a483df"
format_stamp: "Formatted at 2026-09-28 02:09:25 on dist-test-slave-23bp"
server_key: "31b426d16f71cfe9f91c3d27e022f0f4"
server_key_iv: "7b0e8364eb7ae0864c45165949c9365b"
server_key_version: "encryptionkey@0"
I20260928 02:09:25.979033 21233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:09:25.982666 21248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:25.984068 21233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:09:25.984200 21233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "aec5de72bb9e4d37bf649c96a8a483df"
format_stamp: "Formatted at 2026-09-28 02:09:25 on dist-test-slave-23bp"
server_key: "31b426d16f71cfe9f91c3d27e022f0f4"
server_key_iv: "7b0e8364eb7ae0864c45165949c9365b"
server_key_version: "encryptionkey@0"
I20260928 02:09:25.984361 21233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:26.016323 21233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:26.020473 21233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:26.020798 21233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:26.021828 21233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:09:26.023248 21233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:09:26.023324 21233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:26.023412 21233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:09:26.023465 21233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:26.057286 21233 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:34395
I20260928 02:09:26.057335 21361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:34395 every 8 connection(s)
I20260928 02:09:26.058633 21233 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:09:26.062292 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 21233
I20260928 02:09:26.062574 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:09:26.063413 1754 external_mini_cluster.cc:1428] Setting key 1b9e0cfb455be5c3d336170dca08dade
Sep 28 02:09:26 dist-test-slave-23bp krb5kdc[21127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.129@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:09:26.079828 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:26.061477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:33411 (local address 127.1.182.190:42835)
0928 02:09:26.063765 (+ 2288us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:26.063773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:26.063800 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:09:26.065233 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:26.065239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:26.065330 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:26.065448 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:26.066941 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:26.067739 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:26.069488 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:26.069952 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:26.070095 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:26.073921 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:26.073926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:26.073929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:26.073962 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:26.076636 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:26.077368 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:26.077375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:26.077379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:26.077505 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:26.077954 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:26.077960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:26.077964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:26.078242 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:26.078399 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:26.079170 (+ 771us) server_negotiation.cc:299] Negotiation successful
0928 02:09:26.079398 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2156,"thread_start_us":1977,"threads_started":1}
I20260928 02:09:26.080811 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:26.062037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:42835 (local address 127.1.182.129:33411)
0928 02:09:26.062873 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:26.062920 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:26.064954 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:26.065692 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:26.065700 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:26.066205 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:09:26.066766 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:26.066784 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:26.067909 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:26.067913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:26.069348 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:26.069361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:26.070197 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:26.070207 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:26.070447 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:26.071238 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:09:26.071260 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:09:26.073691 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:09:26.076797 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:09:26.076806 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:09:26.076824 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:09:26.077247 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:09:26.077633 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:09:26.077637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:09:26.077641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:09:26.077834 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:09:26.078374 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:09:26.078382 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:09:26.078770 (+ 388us) client_negotiation.cc:769] Sending connection context
0928 02:09:26.079036 (+ 266us) client_negotiation.cc:240] Negotiation successful
0928 02:09:26.079624 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":346,"threads_started":1}
I20260928 02:09:26.082516 21362 heartbeater.cc:347] Connected to a master server at 127.1.182.190:42835
I20260928 02:09:26.082867 21362 heartbeater.cc:464] Registering TS with master...
I20260928 02:09:26.083874 21362 heartbeater.cc:511] Master 127.1.182.190:42835 requested a full tablet report, sending...
I20260928 02:09:26.086585 21172 ts_manager.cc:194] Registered new tserver with Master: aec5de72bb9e4d37bf649c96a8a483df (127.1.182.129:34395)
I20260928 02:09:26.089219 21172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.129@KRBTEST.COM'} at 127.1.182.129:33411
WARNING: no policy specified for kudu/127.1.182.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
Entry for principal kudu/127.1.182.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab.
I20260928 02:09:26.123257 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:42835
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:26.463356 21368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:26.463656 21368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:26.463752 21368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:26.472772 21368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:26.472888 21368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:26.473006 21368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:09:26.483690 21368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.130.keytab
--principal=kudu/127.1.182.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:42835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:26.484982 21368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:26.486243 21368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:26.499944 21374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:26.499969 21376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:26.499924 21373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:26.500468 21368 server_base.cc:1034] running on GCE node
I20260928 02:09:26.501368 21368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:26.502375 21368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:26.503626 21368 hybrid_clock.cc:648] HybridClock initialized: now 1790561366503567 us; error 67 us; skew 500 ppm
I20260928 02:09:26.503950 21368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:26 dist-test-slave-23bp krb5kdc[21127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561366, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:26.506994 21368 init.cc:377] Logged in from keytab as kudu/127.1.182.130@KRBTEST.COM (short username kudu)
I20260928 02:09:26.508455 21368 webserver.cc:460] Webserver started at http://127.1.182.130:43117/ using document root <none> and password file <none>
I20260928 02:09:26.509112 21368 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:26.509209 21368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:26.509474 21368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:26.512504 21368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "54aaee2f10e04582ae1ec9c5a0d64e72"
format_stamp: "Formatted at 2026-09-28 02:09:26 on dist-test-slave-23bp"
server_key: "067fd6e10cbfaf7a3cb225788ce168e9"
server_key_iv: "0c0cf7b663bc80ba45c86fb7579dbad2"
server_key_version: "encryptionkey@0"
I20260928 02:09:26.513265 21368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "54aaee2f10e04582ae1ec9c5a0d64e72"
format_stamp: "Formatted at 2026-09-28 02:09:26 on dist-test-slave-23bp"
server_key: "067fd6e10cbfaf7a3cb225788ce168e9"
server_key_iv: "0c0cf7b663bc80ba45c86fb7579dbad2"
server_key_version: "encryptionkey@0"
I20260928 02:09:26.518356 21368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:09:26.521749 21383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:26.523208 21368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:09:26.523339 21368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "54aaee2f10e04582ae1ec9c5a0d64e72"
format_stamp: "Formatted at 2026-09-28 02:09:26 on dist-test-slave-23bp"
server_key: "067fd6e10cbfaf7a3cb225788ce168e9"
server_key_iv: "0c0cf7b663bc80ba45c86fb7579dbad2"
server_key_version: "encryptionkey@0"
I20260928 02:09:26.523509 21368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:26.546293 21368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:26.550510 21368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:26.550774 21368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:26.551789 21368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:09:26.553134 21368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:09:26.553222 21368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:26.553305 21368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:09:26.553349 21368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:26.587858 21368 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:40765
I20260928 02:09:26.587926 21496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:40765 every 8 connection(s)
I20260928 02:09:26.589318 21368 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:09:26.597956 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 21368
I20260928 02:09:26.598165 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:09:26.598642 1754 external_mini_cluster.cc:1428] Setting key 2c55fccb2695855016980f52a6cb42c3
Sep 28 02:09:26 dist-test-slave-23bp krb5kdc[21127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561366, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.130@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:09:26.611332 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:26.591928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:53495 (local address 127.1.182.190:42835)
0928 02:09:26.592399 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:26.592404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:26.593390 (+ 986us) server_negotiation.cc:407] Connection header received
0928 02:09:26.595383 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:26.595389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:26.595495 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:26.595616 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:26.597157 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:26.597894 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:26.599840 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:26.600325 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:26.600496 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:26.604140 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:26.604149 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:26.604157 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:26.604203 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:26.607854 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:26.608600 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:26.608608 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:26.608614 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:26.608771 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:26.609222 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:26.609230 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:26.609236 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:26.609604 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:26.609719 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:26.610769 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0928 02:09:26.610950 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":252,"threads_started":1}
I20260928 02:09:26.612797 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:26.592412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:42835 (local address 127.1.182.130:53495)
0928 02:09:26.593204 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:26.593242 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:26.595054 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:26.595866 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:26.595874 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:26.596398 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:09:26.596974 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:26.596995 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:26.598044 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:26.598051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:26.599656 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:26.599668 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:26.600620 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:26.600628 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:26.600894 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:26.601613 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:09:26.601636 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:09:26.603878 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:09:26.608006 (+ 4128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:09:26.608013 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:09:26.608026 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:09:26.608455 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:09:26.608886 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:09:26.608889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:09:26.608892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:09:26.609098 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:09:26.609742 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:09:26.609754 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:09:26.610330 (+ 576us) client_negotiation.cc:769] Sending connection context
0928 02:09:26.610674 (+ 344us) client_negotiation.cc:240] Negotiation successful
0928 02:09:26.611304 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":302,"threads_started":1}
I20260928 02:09:26.614327 21497 heartbeater.cc:347] Connected to a master server at 127.1.182.190:42835
I20260928 02:09:26.614651 21497 heartbeater.cc:464] Registering TS with master...
I20260928 02:09:26.615466 21497 heartbeater.cc:511] Master 127.1.182.190:42835 requested a full tablet report, sending...
I20260928 02:09:26.616967 21172 ts_manager.cc:194] Registered new tserver with Master: 54aaee2f10e04582ae1ec9c5a0d64e72 (127.1.182.130:40765)
I20260928 02:09:26.617780 21172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.130@KRBTEST.COM'} at 127.1.182.130:53495
WARNING: no policy specified for kudu/127.1.182.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
Entry for principal kudu/127.1.182.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab.
I20260928 02:09:26.658586 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:42835
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:26.986070 21503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:26.986362 21503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:26.986467 21503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:26.993854 21503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:26.993975 21503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:26.994096 21503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:09:27.004150 21503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.131.keytab
--principal=kudu/127.1.182.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:42835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:27.005439 21503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:27.006724 21503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:27.020103 21508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:27.020063 21511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:27.020136 21509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:27.020877 21503 server_base.cc:1034] running on GCE node
I20260928 02:09:27.021561 21503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:27.022629 21503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:27.023872 21503 hybrid_clock.cc:648] HybridClock initialized: now 1790561367023834 us; error 64 us; skew 500 ppm
I20260928 02:09:27.024211 21503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:27 dist-test-slave-23bp krb5kdc[21127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561367, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:27.027560 21503 init.cc:377] Logged in from keytab as kudu/127.1.182.131@KRBTEST.COM (short username kudu)
I20260928 02:09:27.029114 21503 webserver.cc:460] Webserver started at http://127.1.182.131:37799/ using document root <none> and password file <none>
I20260928 02:09:27.029837 21503 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:27.029987 21503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:27.030330 21503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:27.033481 21503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "0fd7a54c915b492f818391d2386ed3c5"
format_stamp: "Formatted at 2026-09-28 02:09:27 on dist-test-slave-23bp"
server_key: "22f4784e4271b79e85430cbb6e68e12e"
server_key_iv: "dfbe59dcab5ab99697661d67fae73a33"
server_key_version: "encryptionkey@0"
I20260928 02:09:27.034363 21503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "0fd7a54c915b492f818391d2386ed3c5"
format_stamp: "Formatted at 2026-09-28 02:09:27 on dist-test-slave-23bp"
server_key: "22f4784e4271b79e85430cbb6e68e12e"
server_key_iv: "dfbe59dcab5ab99697661d67fae73a33"
server_key_version: "encryptionkey@0"
I20260928 02:09:27.039940 21503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:09:27.043520 21518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:27.044895 21503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 02:09:27.045084 21503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "0fd7a54c915b492f818391d2386ed3c5"
format_stamp: "Formatted at 2026-09-28 02:09:27 on dist-test-slave-23bp"
server_key: "22f4784e4271b79e85430cbb6e68e12e"
server_key_iv: "dfbe59dcab5ab99697661d67fae73a33"
server_key_version: "encryptionkey@0"
I20260928 02:09:27.045287 21503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:27.067864 21503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:27.072005 21503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:27.072324 21503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:27.073398 21503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:09:27.074716 21503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:09:27.074817 21503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:27.074986 21503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:09:27.075083 21503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:27.093287 21362 heartbeater.cc:503] Master 127.1.182.190:42835 was elected leader, sending a full tablet report...
I20260928 02:09:27.109302 21503 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:37993
I20260928 02:09:27.109386 21631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:37993 every 8 connection(s)
I20260928 02:09:27.110718 21503 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:09:27.113827 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 21503
I20260928 02:09:27.114009 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:09:27.114466 1754 external_mini_cluster.cc:1428] Setting key 08de5264685b9db4af6926914442cb04
Sep 28 02:09:27 dist-test-slave-23bp krb5kdc[21127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561367, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.131@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:09:27.131623 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:27.113367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:42573 (local address 127.1.182.190:42835)
0928 02:09:27.113927 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:27.113934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:27.115101 (+ 1167us) server_negotiation.cc:407] Connection header received
0928 02:09:27.117189 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:27.117193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:27.117285 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:27.117412 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:27.118834 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.119656 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.121154 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.121533 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.121635 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:27.125347 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:27.125355 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:27.125361 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:27.125410 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:27.128394 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:27.129353 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:27.129358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:27.129361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:27.129505 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:27.130033 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:27.130036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:27.130038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:27.130311 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:27.130436 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:27.131161 (+ 725us) server_negotiation.cc:299] Negotiation successful
0928 02:09:27.131329 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":443,"threads_started":1}
I20260928 02:09:27.132622 21635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:27.114031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:42835 (local address 127.1.182.131:42573)
0928 02:09:27.114898 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:27.114945 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:27.116894 (+ 1949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:27.117617 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:27.117625 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:27.118121 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:09:27.118691 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:27.118708 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.119790 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.119793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:27.121025 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:27.121037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.121721 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.121725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:27.121914 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:27.122610 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:09:27.122632 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:09:27.125094 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:09:27.128546 (+ 3452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:09:27.128555 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:09:27.128575 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:09:27.129222 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:09:27.129642 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:09:27.129650 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:09:27.129657 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:09:27.129918 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:09:27.130434 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:09:27.130441 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:09:27.130774 (+ 333us) client_negotiation.cc:769] Sending connection context
0928 02:09:27.131050 (+ 276us) client_negotiation.cc:240] Negotiation successful
0928 02:09:27.131525 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":342,"threads_started":1}
I20260928 02:09:27.133915 21632 heartbeater.cc:347] Connected to a master server at 127.1.182.190:42835
I20260928 02:09:27.134233 21632 heartbeater.cc:464] Registering TS with master...
I20260928 02:09:27.134995 21632 heartbeater.cc:511] Master 127.1.182.190:42835 requested a full tablet report, sending...
I20260928 02:09:27.136385 21172 ts_manager.cc:194] Registered new tserver with Master: 0fd7a54c915b492f818391d2386ed3c5 (127.1.182.131:37993)
I20260928 02:09:27.137152 21172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.182.131@KRBTEST.COM'} at 127.1.182.131:42573
I20260928 02:09:27.142104 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:09:27.158545 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:27.146047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48336 (local address 127.1.182.190:42835)
0928 02:09:27.146259 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:27.146269 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:27.146628 (+ 359us) server_negotiation.cc:407] Connection header received
0928 02:09:27.146756 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:27.146761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:27.146821 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:27.146902 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:27.148380 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.149197 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.150533 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.150929 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.151023 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:27.153383 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:27.153388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:27.153391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:27.153417 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:27.156154 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:27.156811 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:27.156815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:27.156817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:27.156936 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:27.157248 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:27.157251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:27.157252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:27.157552 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:27.157723 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:27.158019 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 02:09:27.158200 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:09:27.162532 21172 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:09:27.169400 21172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:09:27.194000 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:27.182502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52480 (local address 127.1.182.129:34395)
0928 02:09:27.185059 (+ 2557us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:27.185070 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:27.185102 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:09:27.185222 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:27.185230 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:27.185505 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:27.186258 (+ 753us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:27.186988 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.188628 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.191210 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.192641 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.192790 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:27.193043 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:27.193464 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 02:09:27.193650 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2350,"thread_start_us":2758,"threads_started":1}
I20260928 02:09:27.194005 21646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:27.181860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:34395 (local address 127.0.0.1:52480)
0928 02:09:27.184742 (+ 2882us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:27.184762 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:27.184873 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:27.185758 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:27.185768 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:27.185796 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:27.186746 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:27.186760 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.188842 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.188848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:27.191014 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:27.191031 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.192861 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.192868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:27.193077 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:27.193154 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 02:09:27.193303 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 02:09:27.193491 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2744,"mutex_wait_us":353,"spinlock_wait_cycles":70272,"thread_start_us":2589,"threads_started":1}
I20260928 02:09:27.194119 21649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:27.184878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:37993 (local address 127.0.0.1:50222)
0928 02:09:27.185876 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:27.185894 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:27.186080 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:27.187246 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:27.187251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:27.187276 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:27.187703 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:27.187715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.189838 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.189843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:27.191867 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:27.191877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.193303 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.193310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:27.193463 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:27.193492 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 02:09:27.193552 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 02:09:27.193698 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":4,"spinlock_wait_cycles":3840,"thread_start_us":868,"threads_started":1}
I20260928 02:09:27.194957 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:27.181830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:40765 (local address 127.0.0.1:48898)
0928 02:09:27.184184 (+ 2354us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:27.184220 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:27.184393 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:27.185936 (+ 1543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:27.185941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:27.185971 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:27.186366 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:27.186381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.188814 (+ 2433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.188819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:27.191401 (+ 2582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:27.191415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.193208 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.193214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:27.193385 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:27.193422 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:09:27.193509 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:09:27.193647 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2111,"mutex_wait_us":128,"thread_start_us":1983,"threads_started":1}
I20260928 02:09:27.195164 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:27.185468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50222 (local address 127.1.182.131:37993)
0928 02:09:27.186609 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:27.186620 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:27.186652 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:09:27.186752 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:27.186759 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:27.186984 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:27.187878 (+ 894us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:27.188160 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.189633 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.192009 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.193061 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.194328 (+ 1267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:27.194575 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:27.194648 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 02:09:27.194809 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":980,"thread_start_us":1719,"threads_started":1}
I20260928 02:09:27.195739 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:27.181934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.1.182.130:40765)
0928 02:09:27.185358 (+ 3424us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:27.185366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:27.185390 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:09:27.185478 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:27.185485 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:27.185722 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:27.186600 (+ 878us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:27.186865 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.188628 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.191578 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.193009 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.193150 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:27.193352 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:27.195323 (+ 1971us) server_negotiation.cc:299] Negotiation successful
0928 02:09:27.195495 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3278,"thread_start_us":3149,"threads_started":1}
I20260928 02:09:27.198374 21566 tablet_service.cc:1467] Processing CreateTablet for tablet 930265e185b341aa918212cae96140c1 (DEFAULT_TABLE table=test-table [id=ab8e6c9a8d9b4752be6798355911b8ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:09:27.198447 21296 tablet_service.cc:1467] Processing CreateTablet for tablet 930265e185b341aa918212cae96140c1 (DEFAULT_TABLE table=test-table [id=ab8e6c9a8d9b4752be6798355911b8ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:09:27.199190 21431 tablet_service.cc:1467] Processing CreateTablet for tablet 930265e185b341aa918212cae96140c1 (DEFAULT_TABLE table=test-table [id=ab8e6c9a8d9b4752be6798355911b8ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:09:27.199931 21296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930265e185b341aa918212cae96140c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:27.200237 21566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930265e185b341aa918212cae96140c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:27.200428 21431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930265e185b341aa918212cae96140c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:27.212114 21651 tablet_bootstrap.cc:492] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5: Bootstrap starting.
I20260928 02:09:27.214915 21652 tablet_bootstrap.cc:492] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72: Bootstrap starting.
I20260928 02:09:27.216917 21651 tablet_bootstrap.cc:654] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:27.218243 21651 log.cc:824] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:27.218477 21652 tablet_bootstrap.cc:654] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:27.219193 21653 tablet_bootstrap.cc:492] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df: Bootstrap starting.
I20260928 02:09:27.219945 21652 log.cc:824] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:27.222719 21653 tablet_bootstrap.cc:654] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:27.222735 21651 tablet_bootstrap.cc:492] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5: No bootstrap required, opened a new log
I20260928 02:09:27.222996 21651 ts_tablet_manager.cc:1397] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260928 02:09:27.223012 21652 tablet_bootstrap.cc:492] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72: No bootstrap required, opened a new log
I20260928 02:09:27.223318 21652 ts_tablet_manager.cc:1397] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260928 02:09:27.223951 21653 log.cc:824] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:27.226924 21653 tablet_bootstrap.cc:492] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df: No bootstrap required, opened a new log
I20260928 02:09:27.227231 21653 ts_tablet_manager.cc:1397] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260928 02:09:27.231128 21651 raft_consensus.cc:348] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54aaee2f10e04582ae1ec9c5a0d64e72" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40765 } } peers { permanent_uuid: "aec5de72bb9e4d37bf649c96a8a483df" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34395 } } peers { permanent_uuid: "0fd7a54c915b492f818391d2386ed3c5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37993 } }
I20260928 02:09:27.231539 21651 raft_consensus.cc:374] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:27.231637 21651 raft_consensus.cc:729] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fd7a54c915b492f818391d2386ed3c5, State: Initialized, Role: FOLLOWER
I20260928 02:09:27.232313 21651 consensus_queue.cc:260] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54aaee2f10e04582ae1ec9c5a0d64e72" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40765 } } peers { permanent_uuid: "aec5de72bb9e4d37bf649c96a8a483df" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34395 } } peers { permanent_uuid: "0fd7a54c915b492f818391d2386ed3c5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37993 } }
I20260928 02:09:27.233743 21652 raft_consensus.cc:348] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54aaee2f10e04582ae1ec9c5a0d64e72" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40765 } } peers { permanent_uuid: "aec5de72bb9e4d37bf649c96a8a483df" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34395 } } peers { permanent_uuid: "0fd7a54c915b492f818391d2386ed3c5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37993 } }
I20260928 02:09:27.234190 21652 raft_consensus.cc:374] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:27.234288 21652 raft_consensus.cc:729] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54aaee2f10e04582ae1ec9c5a0d64e72, State: Initialized, Role: FOLLOWER
I20260928 02:09:27.234769 21632 heartbeater.cc:503] Master 127.1.182.190:42835 was elected leader, sending a full tablet report...
I20260928 02:09:27.235602 21651 ts_tablet_manager.cc:1428] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260928 02:09:27.235555 21652 consensus_queue.cc:260] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54aaee2f10e04582ae1ec9c5a0d64e72" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40765 } } peers { permanent_uuid: "aec5de72bb9e4d37bf649c96a8a483df" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34395 } } peers { permanent_uuid: "0fd7a54c915b492f818391d2386ed3c5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37993 } }
I20260928 02:09:27.237178 21652 ts_tablet_manager.cc:1428] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20260928 02:09:27.237711 21653 raft_consensus.cc:348] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54aaee2f10e04582ae1ec9c5a0d64e72" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40765 } } peers { permanent_uuid: "aec5de72bb9e4d37bf649c96a8a483df" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34395 } } peers { permanent_uuid: "0fd7a54c915b492f818391d2386ed3c5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37993 } }
I20260928 02:09:27.238162 21653 raft_consensus.cc:374] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:27.238291 21653 raft_consensus.cc:729] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec5de72bb9e4d37bf649c96a8a483df, State: Initialized, Role: FOLLOWER
I20260928 02:09:27.238996 21497 heartbeater.cc:503] Master 127.1.182.190:42835 was elected leader, sending a full tablet report...
I20260928 02:09:27.238909 21653 consensus_queue.cc:260] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54aaee2f10e04582ae1ec9c5a0d64e72" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40765 } } peers { permanent_uuid: "aec5de72bb9e4d37bf649c96a8a483df" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34395 } } peers { permanent_uuid: "0fd7a54c915b492f818391d2386ed3c5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37993 } }
I20260928 02:09:27.240743 21653 ts_tablet_manager.cc:1428] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260928 02:09:27.311311 21363 tablet.cc:2369] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:09:27.341439 21498 tablet.cc:2369] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:09:27.361150 21657 raft_consensus.cc:482] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:09:27.361477 21657 raft_consensus.cc:504] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54aaee2f10e04582ae1ec9c5a0d64e72" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40765 } } peers { permanent_uuid: "aec5de72bb9e4d37bf649c96a8a483df" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34395 } } peers { permanent_uuid: "0fd7a54c915b492f818391d2386ed3c5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37993 } }
W20260928 02:09:27.362416 21633 tablet.cc:2369] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:09:27.363646 21657 leader_election.cc:288] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54aaee2f10e04582ae1ec9c5a0d64e72 (127.1.182.130:40765), aec5de72bb9e4d37bf649c96a8a483df (127.1.182.129:34395)
I20260928 02:09:27.369213 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:27.364152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:54217 (local address 127.1.182.130:40765)
0928 02:09:27.364337 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:27.364344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:27.364369 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:09:27.364629 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:27.364634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:27.364703 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:27.364803 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:27.365209 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.365914 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.367560 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.368341 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.368565 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:27.368621 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:27.368862 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 02:09:27.368951 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:09:27.369241 21635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:27.363961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:40765 (local address 127.1.182.131:54217)
0928 02:09:27.364303 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:27.364322 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:27.364456 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:27.364825 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:27.364830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:27.364870 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:27.365066 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:27.365075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.366073 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.366078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:27.367403 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:27.367416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.368508 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.368514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:27.368704 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:27.368739 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:09:27.368834 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 02:09:27.368965 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198}
I20260928 02:09:27.370432 21660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:27.364060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:34395 (local address 127.1.182.131:45117)
0928 02:09:27.364848 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:27.364862 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:27.365025 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:27.365352 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:27.365357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:27.365379 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:27.365696 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:27.365704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.366691 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.366696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:27.368140 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:27.368163 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.369680 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.369686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:27.369830 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:27.369853 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:09:27.369957 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:09:27.370100 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":483,"threads_started":1}
I20260928 02:09:27.370517 21451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930265e185b341aa918212cae96140c1" candidate_uuid: "0fd7a54c915b492f818391d2386ed3c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54aaee2f10e04582ae1ec9c5a0d64e72" is_pre_election: true
I20260928 02:09:27.371031 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:27.364072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:45117 (local address 127.1.182.129:34395)
0928 02:09:27.364308 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:27.364318 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:27.364961 (+ 643us) server_negotiation.cc:407] Connection header received
0928 02:09:27.365176 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:27.365180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:27.365238 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:27.365339 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:27.365840 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.366533 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.368299 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.369477 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.369638 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:27.369708 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:27.370670 (+ 962us) server_negotiation.cc:299] Negotiation successful
0928 02:09:27.370809 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:09:27.371124 21451 raft_consensus.cc:2454] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fd7a54c915b492f818391d2386ed3c5 in term 0.
I20260928 02:09:27.371832 21520 leader_election.cc:302] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fd7a54c915b492f818391d2386ed3c5, 54aaee2f10e04582ae1ec9c5a0d64e72; no voters:
I20260928 02:09:27.371958 21316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930265e185b341aa918212cae96140c1" candidate_uuid: "0fd7a54c915b492f818391d2386ed3c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec5de72bb9e4d37bf649c96a8a483df" is_pre_election: true
I20260928 02:09:27.372216 21657 raft_consensus.cc:2781] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:09:27.372373 21657 raft_consensus.cc:482] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:09:27.372465 21657 raft_consensus.cc:3037] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:27.372539 21316 raft_consensus.cc:2454] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fd7a54c915b492f818391d2386ed3c5 in term 0.
I20260928 02:09:27.374989 21657 raft_consensus.cc:504] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54aaee2f10e04582ae1ec9c5a0d64e72" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40765 } } peers { permanent_uuid: "aec5de72bb9e4d37bf649c96a8a483df" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34395 } } peers { permanent_uuid: "0fd7a54c915b492f818391d2386ed3c5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37993 } }
I20260928 02:09:27.375706 21657 leader_election.cc:288] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 [CANDIDATE]: Term 1 election: Requested vote from peers 54aaee2f10e04582ae1ec9c5a0d64e72 (127.1.182.130:40765), aec5de72bb9e4d37bf649c96a8a483df (127.1.182.129:34395)
I20260928 02:09:27.376124 21451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930265e185b341aa918212cae96140c1" candidate_uuid: "0fd7a54c915b492f818391d2386ed3c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54aaee2f10e04582ae1ec9c5a0d64e72"
I20260928 02:09:27.376338 21451 raft_consensus.cc:3037] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:27.376276 21316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930265e185b341aa918212cae96140c1" candidate_uuid: "0fd7a54c915b492f818391d2386ed3c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec5de72bb9e4d37bf649c96a8a483df"
I20260928 02:09:27.376529 21316 raft_consensus.cc:3037] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:27.378521 21451 raft_consensus.cc:2454] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fd7a54c915b492f818391d2386ed3c5 in term 1.
I20260928 02:09:27.378748 21316 raft_consensus.cc:2454] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fd7a54c915b492f818391d2386ed3c5 in term 1.
I20260928 02:09:27.378908 21520 leader_election.cc:302] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fd7a54c915b492f818391d2386ed3c5, 54aaee2f10e04582ae1ec9c5a0d64e72; no voters:
I20260928 02:09:27.379222 21657 raft_consensus.cc:2781] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:27.379657 21657 raft_consensus.cc:686] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 [term 1 LEADER]: Becoming Leader. State: Replica: 0fd7a54c915b492f818391d2386ed3c5, State: Running, Role: LEADER
I20260928 02:09:27.380188 21657 consensus_queue.cc:237] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54aaee2f10e04582ae1ec9c5a0d64e72" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40765 } } peers { permanent_uuid: "aec5de72bb9e4d37bf649c96a8a483df" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34395 } } peers { permanent_uuid: "0fd7a54c915b492f818391d2386ed3c5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37993 } }
I20260928 02:09:27.384898 21172 catalog_manager.cc:5495] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fd7a54c915b492f818391d2386ed3c5 (127.1.182.131). New cstate: current_term: 1 leader_uuid: "0fd7a54c915b492f818391d2386ed3c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54aaee2f10e04582ae1ec9c5a0d64e72" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aec5de72bb9e4d37bf649c96a8a483df" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fd7a54c915b492f818391d2386ed3c5" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 37993 } health_report { overall_health: HEALTHY } } }
I20260928 02:09:27.433014 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:27.427045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50226 (local address 127.1.182.131:37993)
0928 02:09:27.427276 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:27.427283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:27.427520 (+ 237us) server_negotiation.cc:407] Connection header received
0928 02:09:27.427806 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:27.427810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:27.427904 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:27.428075 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:09:27.428549 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.429284 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.430705 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.431213 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.431346 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:27.431885 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:09:27.432423 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:09:27.432590 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:27.432670 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 02:09:27.432750 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:09:27.441331 21316 raft_consensus.cc:1261] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df [term 1 FOLLOWER]: Refusing update from remote peer 0fd7a54c915b492f818391d2386ed3c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:09:27.441363 21451 raft_consensus.cc:1261] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72 [term 1 FOLLOWER]: Refusing update from remote peer 0fd7a54c915b492f818391d2386ed3c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:09:27.442329 21661 consensus_queue.cc:1035] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec5de72bb9e4d37bf649c96a8a483df" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 34395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:09:27.442693 21657 consensus_queue.cc:1035] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54aaee2f10e04582ae1ec9c5a0d64e72" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:09:27.453871 21664 mvcc.cc:205] Tried to move back new op lower bound from 7334139361029537792 to 7334139360792764416. Current Snapshot: MvccSnapshot[applied={T|T < 7334139361029537792}]
I20260928 02:09:27.456655 21666 mvcc.cc:205] Tried to move back new op lower bound from 7334139361029537792 to 7334139360792764416. Current Snapshot: MvccSnapshot[applied={T|T < 7334139361029537792}]
I20260928 02:09:27.471719 21171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:09:27.472059 21171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:09:27.476637 21171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e989430d06514bc7b22e67f7f0227cbe: Sending DeleteTablet for 3 replicas of tablet 930265e185b341aa918212cae96140c1
I20260928 02:09:27.478109 21431 tablet_service.cc:1514] Processing DeleteTablet for tablet 930265e185b341aa918212cae96140c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:09:27 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:48898
I20260928 02:09:27.478155 21296 tablet_service.cc:1514] Processing DeleteTablet for tablet 930265e185b341aa918212cae96140c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:09:27 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:52480
I20260928 02:09:27.478942 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 21233
I20260928 02:09:27.479262 21566 tablet_service.cc:1514] Processing DeleteTablet for tablet 930265e185b341aa918212cae96140c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:09:27 UTC) from {username='kudu', principal='kudu/127.1.182.190@KRBTEST.COM'} at 127.0.0.1:50222
I20260928 02:09:27.492992 21673 tablet_replica.cc:330] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72: stopping tablet replica
I20260928 02:09:27.493592 21673 raft_consensus.cc:2229] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:09:27.494197 21673 raft_consensus.cc:2258] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:09:27.495229 21674 tablet_replica.cc:330] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df: stopping tablet replica
I20260928 02:09:27.495800 21674 raft_consensus.cc:2229] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:09:27.496423 21674 raft_consensus.cc:2258] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:09:27.499202 21674 ts_tablet_manager.cc:1905] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:09:27.499202 21673 ts_tablet_manager.cc:1905] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 02:09:27 dist-test-slave-23bp krb5kdc[21127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.129@KRBTEST.COM
I20260928 02:09:27.503686 21675 tablet_replica.cc:330] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5: stopping tablet replica
I20260928 02:09:27.504161 21675 raft_consensus.cc:2229] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:09:27.504907 21675 raft_consensus.cc:2258] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:09:27.506712 21674 ts_tablet_manager.cc:1918] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:09:27.506908 21674 log.cc:1196] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-0/wal/wals/930265e185b341aa918212cae96140c1
I20260928 02:09:27.507525 21674 ts_tablet_manager.cc:1939] T 930265e185b341aa918212cae96140c1 P aec5de72bb9e4d37bf649c96a8a483df: Deleting consensus metadata
I20260928 02:09:27.509423 21675 ts_tablet_manager.cc:1905] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:09:27.509533 21156 catalog_manager.cc:4841] TS aec5de72bb9e4d37bf649c96a8a483df (127.1.182.129:34395): tablet 930265e185b341aa918212cae96140c1 (table test-table [id=ab8e6c9a8d9b4752be6798355911b8ab]) successfully deleted
I20260928 02:09:27.511775 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:27.480232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52492 (local address 127.1.182.129:34395)
0928 02:09:27.480479 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:27.480488 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:27.490711 (+ 10223us) server_negotiation.cc:407] Connection header received
0928 02:09:27.490926 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:27.490932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:27.491031 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:27.491169 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:27.492970 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.494149 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.496813 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.497495 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.497627 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:27.503284 (+ 5657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:27.503299 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:27.503322 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:27.503376 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:27.507870 (+ 4494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:27.508791 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:27.508800 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:27.508804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:27.508998 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:27.509468 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:27.509473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:27.509477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:27.510098 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:27.510388 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:27.511037 (+ 649us) server_negotiation.cc:299] Negotiation successful
0928 02:09:27.511301 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:09:27.513572 21357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:27.516737 21673 ts_tablet_manager.cc:1918] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:09:27.517001 21673 log.cc:1196] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/930265e185b341aa918212cae96140c1
I20260928 02:09:27.517591 21673 ts_tablet_manager.cc:1939] T 930265e185b341aa918212cae96140c1 P 54aaee2f10e04582ae1ec9c5a0d64e72: Deleting consensus metadata
I20260928 02:09:27.518090 21675 ts_tablet_manager.cc:1918] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:09:27.518275 21675 log.cc:1196] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/930265e185b341aa918212cae96140c1
I20260928 02:09:27.518903 21675 ts_tablet_manager.cc:1939] T 930265e185b341aa918212cae96140c1 P 0fd7a54c915b492f818391d2386ed3c5: Deleting consensus metadata
I20260928 02:09:27.519837 21157 catalog_manager.cc:4841] TS 54aaee2f10e04582ae1ec9c5a0d64e72 (127.1.182.130:40765): tablet 930265e185b341aa918212cae96140c1 (table test-table [id=ab8e6c9a8d9b4752be6798355911b8ab]) successfully deleted
I20260928 02:09:27.520757 21156 catalog_manager.cc:4841] TS 0fd7a54c915b492f818391d2386ed3c5 (127.1.182.131:37993): tablet 930265e185b341aa918212cae96140c1 (table test-table [id=ab8e6c9a8d9b4752be6798355911b8ab]) successfully deleted
I20260928 02:09:27.659338 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 21233
I20260928 02:09:27.677433 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 21368
Sep 28 02:09:27 dist-test-slave-23bp krb5kdc[21127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.130@KRBTEST.COM
I20260928 02:09:27.694422 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:27.678274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48910 (local address 127.1.182.130:40765)
0928 02:09:27.678494 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:27.678501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:27.678524 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:09:27.678702 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:27.678707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:27.678802 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:27.678917 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:27.680434 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.681587 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.682821 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.683508 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.683640 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:27.687752 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:27.687771 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:27.687792 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:27.687835 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:27.691589 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:27.692237 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:27.692246 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:27.692250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:27.692402 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:27.692751 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:27.692757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:27.692760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:27.693254 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:27.693549 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:27.693765 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 02:09:27.693999 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:09:27.695173 21492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:27.786410 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 21368
I20260928 02:09:27.810348 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 21503
Sep 28 02:09:27 dist-test-slave-23bp krb5kdc[21127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.131@KRBTEST.COM
I20260928 02:09:27.828531 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:27.811187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50236 (local address 127.1.182.131:37993)
0928 02:09:27.811358 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:27.811362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:27.811435 (+ 73us) server_negotiation.cc:407] Connection header received
0928 02:09:27.811627 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:27.811631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:27.811705 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:27.811796 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:27.813667 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.814460 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.816413 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:27.816873 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:27.816971 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:27.822412 (+ 5441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:27.822426 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:27.822445 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:27.822487 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:27.825553 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:27.826298 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:27.826305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:27.826308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:27.826474 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:27.826864 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:27.826877 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:27.826879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:27.827320 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:27.827627 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:27.827931 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 02:09:27.828175 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:09:27.829279 21627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:27.951366 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 21503
I20260928 02:09:27.971061 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 21140
I20260928 02:09:27.972287 21202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:28.083853 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 21140
2026-09-28T02:09:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4514 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T02:09:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:09:28Z Disabled control of system clock
I20260928 02:09:28.160158 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:36705
--webserver_interface=127.1.182.190
--webserver_port=0
--builtin_ntp_servers=127.1.182.148:43265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:36705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:09:28.490513 21684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:28.490825 21684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:28.490907 21684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:28.498819 21684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:09:28.498996 21684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:28.499135 21684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:28.499282 21684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:09:28.499359 21684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:09:28.509227 21684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:36705
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:36705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:28.510555 21684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:28.512483 21684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:28.524507 21692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:28.524468 21690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:28.524535 21689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:28.525420 21684 server_base.cc:1034] running on GCE node
I20260928 02:09:28.526253 21684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:28.527762 21684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:28.528982 21684 hybrid_clock.cc:648] HybridClock initialized: now 1790561368528941 us; error 27 us; skew 500 ppm
I20260928 02:09:28.529349 21684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:09:28.531647 21684 webserver.cc:460] Webserver started at http://127.1.182.190:34415/ using document root <none> and password file <none>
I20260928 02:09:28.532354 21684 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:28.532475 21684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:28.532800 21684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:28.535830 21684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "7bb488d8254d4917a27463727237c1a6"
format_stamp: "Formatted at 2026-09-28 02:09:28 on dist-test-slave-23bp"
server_key: "4a98d3ad7ace6082935d9c38e8755db0"
server_key_iv: "7c1e31bfbd03fdefff9a96feb9609893"
server_key_version: "encryptionkey@0"
I20260928 02:09:28.536628 21684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "7bb488d8254d4917a27463727237c1a6"
format_stamp: "Formatted at 2026-09-28 02:09:28 on dist-test-slave-23bp"
server_key: "4a98d3ad7ace6082935d9c38e8755db0"
server_key_iv: "7c1e31bfbd03fdefff9a96feb9609893"
server_key_version: "encryptionkey@0"
I20260928 02:09:28.541872 21684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:09:28.545502 21698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:28.546978 21684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:09:28.547207 21684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "7bb488d8254d4917a27463727237c1a6"
format_stamp: "Formatted at 2026-09-28 02:09:28 on dist-test-slave-23bp"
server_key: "4a98d3ad7ace6082935d9c38e8755db0"
server_key_iv: "7c1e31bfbd03fdefff9a96feb9609893"
server_key_version: "encryptionkey@0"
I20260928 02:09:28.547446 21684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:28.561738 21684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:28.562613 21684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:28.562911 21684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:28.581609 21684 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:36705
I20260928 02:09:28.581629 21750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:36705 every 8 connection(s)
I20260928 02:09:28.583047 21684 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:09:28.586889 21752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:28.593304 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 21684
I20260928 02:09:28.593513 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:09:28.593950 1754 external_mini_cluster.cc:1428] Setting key 60b2f98750e44aa8b977b612c25f779a
I20260928 02:09:28.597975 21752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6: Bootstrap starting.
I20260928 02:09:28.601202 21752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:28.602270 21752 log.cc:824] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:28.605752 21752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6: No bootstrap required, opened a new log
I20260928 02:09:28.608709 21755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:28.595939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57618 (local address 127.1.182.190:36705)
0928 02:09:28.596915 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:28.596935 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:28.596976 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:09:28.598792 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:28.598816 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:28.599384 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:28.599787 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:28.600372 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:28.601755 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:28.603474 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:28.604494 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:28.604694 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:28.605768 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:28.605781 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:28.605801 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:28.606555 (+ 754us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:28.606660 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:28.606686 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:28.606799 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:28.607023 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 02:09:28.607389 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":321,"threads_started":1}
I20260928 02:09:28.614907 21752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb488d8254d4917a27463727237c1a6" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36705 } }
I20260928 02:09:28.615286 21752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:28.615363 21752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bb488d8254d4917a27463727237c1a6, State: Initialized, Role: FOLLOWER
I20260928 02:09:28.615883 21752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb488d8254d4917a27463727237c1a6" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36705 } }
I20260928 02:09:28.616089 21752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:09:28.616214 21752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:09:28.616340 21752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:28.618400 21752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb488d8254d4917a27463727237c1a6" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36705 } }
I20260928 02:09:28.618826 21752 leader_election.cc:302] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bb488d8254d4917a27463727237c1a6; no voters:
I20260928 02:09:28.619437 21752 leader_election.cc:288] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:09:28.619596 21757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:28.620424 21757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6 [term 1 LEADER]: Becoming Leader. State: Replica: 7bb488d8254d4917a27463727237c1a6, State: Running, Role: LEADER
I20260928 02:09:28.620915 21757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb488d8254d4917a27463727237c1a6" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36705 } }
I20260928 02:09:28.621444 21752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:09:28.623898 21758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bb488d8254d4917a27463727237c1a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb488d8254d4917a27463727237c1a6" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36705 } } }
I20260928 02:09:28.623776 21759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bb488d8254d4917a27463727237c1a6. Latest consensus state: current_term: 1 leader_uuid: "7bb488d8254d4917a27463727237c1a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb488d8254d4917a27463727237c1a6" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36705 } } }
I20260928 02:09:28.624296 21758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:28.624331 21759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:28.625968 21766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:09:28.631949 21766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:09:28.640995 21766 catalog_manager.cc:1347] Generated new cluster ID: 250556ac6e1d4cb9afbeecb6170fc636
I20260928 02:09:28.641125 21766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:09:28.650262 21766 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:09:28.651564 21766 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:09:28.662074 21766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6: Generated new TSK 0
I20260928 02:09:28.662940 21766 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:09:28.667197 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.129:0
--local_ip_for_outbound_sockets=127.1.182.129
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:36705
--builtin_ntp_servers=127.1.182.148:43265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:09:29.010371 21776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:29.010684 21776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:29.010771 21776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:29.018311 21776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:29.018417 21776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:29.018528 21776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.129
I20260928 02:09:29.028170 21776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.182.129
--webserver_port=0
--tserver_master_addrs=127.1.182.190:36705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.129
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:29.029409 21776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:29.031248 21776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:29.044915 21784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:29.044883 21782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:29.045070 21781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:29.045773 21776 server_base.cc:1034] running on GCE node
I20260928 02:09:29.046352 21776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:29.047304 21776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:29.048502 21776 hybrid_clock.cc:648] HybridClock initialized: now 1790561369048462 us; error 55 us; skew 500 ppm
I20260928 02:09:29.048777 21776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:09:29.051028 21776 webserver.cc:460] Webserver started at http://127.1.182.129:34889/ using document root <none> and password file <none>
I20260928 02:09:29.051724 21776 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:29.051810 21776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:29.052134 21776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:29.054867 21776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-0/data/instance:
uuid: "98bf90f55c734da6a8573ead733403b6"
format_stamp: "Formatted at 2026-09-28 02:09:29 on dist-test-slave-23bp"
server_key: "9006badc79219be0df2c8f503ebae926"
server_key_iv: "544704e745c1d63c775108884bf80d50"
server_key_version: "encryptionkey@0"
I20260928 02:09:29.055689 21776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance:
uuid: "98bf90f55c734da6a8573ead733403b6"
format_stamp: "Formatted at 2026-09-28 02:09:29 on dist-test-slave-23bp"
server_key: "9006badc79219be0df2c8f503ebae926"
server_key_iv: "544704e745c1d63c775108884bf80d50"
server_key_version: "encryptionkey@0"
I20260928 02:09:29.060434 21776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:09:29.063793 21790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:29.065222 21776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:09:29.065392 21776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-0/wal
uuid: "98bf90f55c734da6a8573ead733403b6"
format_stamp: "Formatted at 2026-09-28 02:09:29 on dist-test-slave-23bp"
server_key: "9006badc79219be0df2c8f503ebae926"
server_key_iv: "544704e745c1d63c775108884bf80d50"
server_key_version: "encryptionkey@0"
I20260928 02:09:29.065629 21776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:29.103999 21776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:29.104885 21776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:29.105198 21776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:29.106174 21776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:09:29.107831 21776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:09:29.107952 21776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:29.108083 21776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:09:29.108180 21776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:29.141768 21776 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.129:45893
I20260928 02:09:29.141817 21902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.129:45893 every 8 connection(s)
I20260928 02:09:29.143134 21776 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-0/data/info.pb
I20260928 02:09:29.146476 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 21776
I20260928 02:09:29.146669 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-0/wal/instance
I20260928 02:09:29.147254 1754 external_mini_cluster.cc:1428] Setting key ba2c90f6530bb1caf506a57a1490c30c
I20260928 02:09:29.150383 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.130:0
--local_ip_for_outbound_sockets=127.1.182.130
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:36705
--builtin_ntp_servers=127.1.182.148:43265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:09:29.157809 21905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:29.145640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:51279 (local address 127.1.182.190:36705)
0928 02:09:29.146139 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:29.146147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:29.147440 (+ 1293us) server_negotiation.cc:407] Connection header received
0928 02:09:29.149702 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:29.149706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:29.149767 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:29.149857 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:29.151330 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:29.152162 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:29.154701 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:29.155187 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:29.155287 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:29.157053 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:29.157057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:29.157060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:29.157079 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:29.157123 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:29.157135 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:29.157205 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:29.157471 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 02:09:29.157551 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":250,"threads_started":1}
I20260928 02:09:29.159397 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:29.146132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:36705 (local address 127.1.182.129:51279)
0928 02:09:29.147264 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:29.147321 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:29.149393 (+ 2072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:29.150133 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:29.150147 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:29.150279 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:09:29.151142 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:29.151171 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:29.152381 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:29.152390 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:29.154537 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:29.154566 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:29.155423 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:29.155432 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:29.155795 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:29.156602 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:09:29.156608 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:09:29.156663 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:09:29.156848 (+ 185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:09:29.157240 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:09:29.157253 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:09:29.157264 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 02:09:29.157458 (+ 194us) client_negotiation.cc:240] Negotiation successful
0928 02:09:29.157916 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":303,"threads_started":1}
I20260928 02:09:29.161164 21903 heartbeater.cc:347] Connected to a master server at 127.1.182.190:36705
I20260928 02:09:29.161516 21903 heartbeater.cc:464] Registering TS with master...
I20260928 02:09:29.162420 21903 heartbeater.cc:511] Master 127.1.182.190:36705 requested a full tablet report, sending...
I20260928 02:09:29.164829 21715 ts_manager.cc:194] Registered new tserver with Master: 98bf90f55c734da6a8573ead733403b6 (127.1.182.129:45893)
I20260928 02:09:29.166944 21715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.182.129:51279
W20260928 02:09:29.484944 21907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:29.485239 21907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:29.485332 21907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:29.492698 21907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:29.492802 21907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:29.492909 21907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.130
I20260928 02:09:29.501950 21907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.182.130
--webserver_port=0
--tserver_master_addrs=127.1.182.190:36705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.130
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:29.503250 21907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:29.504977 21907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:29.518133 21913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:29.518383 21912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:29.518466 21915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:29.519150 21907 server_base.cc:1034] running on GCE node
I20260928 02:09:29.519771 21907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:29.520771 21907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:29.521984 21907 hybrid_clock.cc:648] HybridClock initialized: now 1790561369521936 us; error 64 us; skew 500 ppm
I20260928 02:09:29.522315 21907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:09:29.524681 21907 webserver.cc:460] Webserver started at http://127.1.182.130:45997/ using document root <none> and password file <none>
I20260928 02:09:29.525467 21907 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:29.525564 21907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:29.525875 21907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:29.528887 21907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-1/data/instance:
uuid: "5241e13727f5494c975691539f2def63"
format_stamp: "Formatted at 2026-09-28 02:09:29 on dist-test-slave-23bp"
server_key: "0d8f51f22aba00a972baf15d0ccb589c"
server_key_iv: "85e2caba76e30cd569589d34648e7be5"
server_key_version: "encryptionkey@0"
I20260928 02:09:29.529667 21907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance:
uuid: "5241e13727f5494c975691539f2def63"
format_stamp: "Formatted at 2026-09-28 02:09:29 on dist-test-slave-23bp"
server_key: "0d8f51f22aba00a972baf15d0ccb589c"
server_key_iv: "85e2caba76e30cd569589d34648e7be5"
server_key_version: "encryptionkey@0"
I20260928 02:09:29.534862 21907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:09:29.538378 21921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:29.539948 21907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:09:29.540117 21907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-1/wal
uuid: "5241e13727f5494c975691539f2def63"
format_stamp: "Formatted at 2026-09-28 02:09:29 on dist-test-slave-23bp"
server_key: "0d8f51f22aba00a972baf15d0ccb589c"
server_key_iv: "85e2caba76e30cd569589d34648e7be5"
server_key_version: "encryptionkey@0"
I20260928 02:09:29.540346 21907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:29.553637 21907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:29.554443 21907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:29.554780 21907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:29.555859 21907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:09:29.557278 21907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:09:29.557406 21907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:29.557547 21907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:09:29.557644 21907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:29.591651 21907 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.130:35793
I20260928 02:09:29.591673 22033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.130:35793 every 8 connection(s)
I20260928 02:09:29.593220 21907 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-1/data/info.pb
I20260928 02:09:29.600529 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 21907
I20260928 02:09:29.600734 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-1/wal/instance
I20260928 02:09:29.601259 1754 external_mini_cluster.cc:1428] Setting key 27a57bd800902a835890db7726e172b6
I20260928 02:09:29.604141 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.182.131:0
--local_ip_for_outbound_sockets=127.1.182.131
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:36705
--builtin_ntp_servers=127.1.182.148:43265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:09:29.607787 21905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:29.596131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.130:51733 (local address 127.1.182.190:36705)
0928 02:09:29.596302 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:29.596306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:29.597918 (+ 1612us) server_negotiation.cc:407] Connection header received
0928 02:09:29.599806 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:29.599812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:29.599907 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:29.600030 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:29.601533 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:29.602480 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:29.604587 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:29.605328 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:29.605494 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:29.606815 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:29.606823 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:29.606830 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:29.606854 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:29.606895 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:29.606913 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:29.607008 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:29.607292 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 02:09:29.607410 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:09:29.608662 22036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:29.596815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:36705 (local address 127.1.182.130:51733)
0928 02:09:29.597756 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:29.597791 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:29.599533 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:29.600265 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:29.600274 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:29.600362 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:09:29.601241 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:29.601260 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:29.602646 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:29.602652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:29.604440 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:29.604454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:29.605534 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:29.605539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:29.605755 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:29.606496 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:09:29.606498 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:09:29.606539 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:09:29.606625 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:09:29.607010 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:09:29.607018 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:09:29.607024 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:09:29.607248 (+ 224us) client_negotiation.cc:240] Negotiation successful
0928 02:09:29.607548 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":295,"threads_started":1}
I20260928 02:09:29.610117 22034 heartbeater.cc:347] Connected to a master server at 127.1.182.190:36705
I20260928 02:09:29.610472 22034 heartbeater.cc:464] Registering TS with master...
I20260928 02:09:29.611402 22034 heartbeater.cc:511] Master 127.1.182.190:36705 requested a full tablet report, sending...
I20260928 02:09:29.612871 21715 ts_manager.cc:194] Registered new tserver with Master: 5241e13727f5494c975691539f2def63 (127.1.182.130:35793)
I20260928 02:09:29.613677 21715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.182.130:51733
W20260928 02:09:29.936264 22037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:29.936553 22037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:29.936734 22037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:29.944343 22037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:29.944500 22037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:29.944674 22037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.182.131
I20260928 02:09:29.954138 22037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:43265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.182.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.182.131
--webserver_port=0
--tserver_master_addrs=127.1.182.190:36705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.182.131
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:29.955551 22037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:29.957335 22037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:29.970780 22042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:29.970875 22043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:29.970847 22045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:29.971752 22037 server_base.cc:1034] running on GCE node
I20260928 02:09:29.972307 22037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:29.973201 22037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:29.974402 22037 hybrid_clock.cc:648] HybridClock initialized: now 1790561369974366 us; error 30 us; skew 500 ppm
I20260928 02:09:29.974689 22037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:09:29.977061 22037 webserver.cc:460] Webserver started at http://127.1.182.131:42975/ using document root <none> and password file <none>
I20260928 02:09:29.977721 22037 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:29.977807 22037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:29.978083 22037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:29.981069 22037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-2/data/instance:
uuid: "1ea270e37cce4d5883caeded1accf4e8"
format_stamp: "Formatted at 2026-09-28 02:09:29 on dist-test-slave-23bp"
server_key: "30de273dd3a7a623b38cb5d09dc0ecfc"
server_key_iv: "33d772579ef10a7fea0759d41cc66d90"
server_key_version: "encryptionkey@0"
I20260928 02:09:29.981858 22037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance:
uuid: "1ea270e37cce4d5883caeded1accf4e8"
format_stamp: "Formatted at 2026-09-28 02:09:29 on dist-test-slave-23bp"
server_key: "30de273dd3a7a623b38cb5d09dc0ecfc"
server_key_iv: "33d772579ef10a7fea0759d41cc66d90"
server_key_version: "encryptionkey@0"
I20260928 02:09:29.986932 22037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:09:29.990422 22051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:29.991822 22037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:09:29.991948 22037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-2/wal
uuid: "1ea270e37cce4d5883caeded1accf4e8"
format_stamp: "Formatted at 2026-09-28 02:09:29 on dist-test-slave-23bp"
server_key: "30de273dd3a7a623b38cb5d09dc0ecfc"
server_key_iv: "33d772579ef10a7fea0759d41cc66d90"
server_key_version: "encryptionkey@0"
I20260928 02:09:29.992129 22037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:30.013125 22037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:30.013990 22037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:30.014263 22037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:30.015303 22037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:09:30.016793 22037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:09:30.016880 22037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:30.016983 22037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:09:30.017042 22037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:30.050767 22037 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.131:36185
I20260928 02:09:30.050809 22163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.131:36185 every 8 connection(s)
I20260928 02:09:30.052269 22037 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-2/data/info.pb
I20260928 02:09:30.061707 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 22037
I20260928 02:09:30.061885 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-2/wal/instance
I20260928 02:09:30.062451 1754 external_mini_cluster.cc:1428] Setting key 1af40d17f98d8c0999a69ffab7eac6d6
I20260928 02:09:30.065372 21905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.054861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.131:55741 (local address 127.1.182.190:36705)
0928 02:09:30.055060 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:30.055113 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:30.056542 (+ 1429us) server_negotiation.cc:407] Connection header received
0928 02:09:30.058319 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:30.058327 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:30.058423 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:30.058536 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:30.059653 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.060415 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.061969 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.062620 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.062766 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:30.064475 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:30.064481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:30.064486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:30.064505 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:30.064541 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:30.064571 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:30.064667 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:30.064924 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 02:09:30.065022 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 02:09:30.066334 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.055493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:36705 (local address 127.1.182.131:55741)
0928 02:09:30.056379 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:30.056420 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:30.058039 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:30.058763 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:30.058771 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:30.058858 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:09:30.059491 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:30.059507 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.060569 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.060576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:30.061800 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:30.061811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.062822 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.062827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:30.063180 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:30.064105 (+ 925us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:09:30.064112 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:09:30.064166 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:09:30.064273 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:09:30.064705 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:09:30.064720 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:09:30.064726 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:09:30.064866 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 02:09:30.065248 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":263,"threads_started":1}
I20260928 02:09:30.067842 22164 heartbeater.cc:347] Connected to a master server at 127.1.182.190:36705
I20260928 02:09:30.068181 22164 heartbeater.cc:464] Registering TS with master...
I20260928 02:09:30.068903 22164 heartbeater.cc:511] Master 127.1.182.190:36705 requested a full tablet report, sending...
I20260928 02:09:30.070356 21715 ts_manager.cc:194] Registered new tserver with Master: 1ea270e37cce4d5883caeded1accf4e8 (127.1.182.131:36185)
I20260928 02:09:30.071197 21715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.182.131:55741
I20260928 02:09:30.077382 1754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:09:30.086856 21905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.081292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57622 (local address 127.1.182.190:36705)
0928 02:09:30.081523 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:30.081530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:30.081827 (+ 297us) server_negotiation.cc:407] Connection header received
0928 02:09:30.081955 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:30.081959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:30.082018 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:30.082101 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:30.082630 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.083381 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.084777 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.085168 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.085258 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:30.086190 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:30.086194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:30.086197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:30.086213 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:30.086244 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:30.086255 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:30.086321 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:30.086456 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 02:09:30.086533 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:09:30.090400 21715 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:09:30.097295 21715 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:09:30.144330 22177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.109045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53830 (local address 127.1.182.130:35793)
0928 02:09:30.135559 (+ 26514us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:30.135569 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:30.135593 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:09:30.135681 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:30.135687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:30.135945 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:30.136164 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:30.136684 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.138161 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.140910 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.142504 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.142644 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:30.142898 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:30.143803 (+ 905us) server_negotiation.cc:299] Negotiation successful
0928 02:09:30.143972 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26320,"thread_start_us":26207,"threads_started":1}
I20260928 02:09:30.144353 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.109316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:36185 (local address 127.0.0.1:41180)
0928 02:09:30.135350 (+ 26034us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:30.135369 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:30.135513 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:30.136708 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:30.136712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:30.136738 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:30.137106 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:30.137117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.138965 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.138970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:30.140777 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:30.140794 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.143088 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.143093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:30.143409 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:30.143537 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 02:09:30.143667 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 02:09:30.143932 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25896,"mutex_wait_us":75,"thread_start_us":25581,"threads_started":1}
I20260928 02:09:30.144995 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.109830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:45893 (local address 127.0.0.1:52334)
0928 02:09:30.136552 (+ 26722us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:30.136566 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:30.136706 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:30.137504 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:30.137538 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:30.137562 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:30.137912 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:30.137924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.139802 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.139808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:30.141964 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:30.141979 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.144220 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.144228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:30.144424 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:30.144458 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:09:30.144562 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:09:30.144687 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26603,"mutex_wait_us":1,"thread_start_us":25346,"threads_started":1}
I20260928 02:09:30.145280 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.109947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.1.182.131:36185)
0928 02:09:30.129773 (+ 19826us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:30.129787 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:30.135505 (+ 5718us) server_negotiation.cc:407] Connection header received
0928 02:09:30.135659 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:30.135669 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:30.135945 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:30.136161 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:30.137303 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.138823 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.141036 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.142504 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.142644 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:30.142930 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:30.144622 (+ 1692us) server_negotiation.cc:299] Negotiation successful
0928 02:09:30.144791 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19573,"thread_start_us":19859,"threads_started":1}
I20260928 02:09:30.145910 22176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.108748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:35793 (local address 127.0.0.1:53830)
0928 02:09:30.134282 (+ 25534us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:30.134323 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:30.134518 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:30.136181 (+ 1663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:30.136187 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:30.136210 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:30.136502 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:30.136509 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.138330 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.138335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:30.140588 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:30.140602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.143043 (+ 2441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.143083 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:30.143412 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:30.143537 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 02:09:30.143667 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 02:09:30.145661 (+ 1994us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25246,"mutex_wait_us":107,"thread_start_us":25104,"threads_started":1}
I20260928 02:09:30.146194 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.110478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.1.182.129:45893)
0928 02:09:30.134434 (+ 23956us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:30.134448 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:30.136717 (+ 2269us) server_negotiation.cc:407] Connection header received
0928 02:09:30.136897 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:30.136905 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:30.137169 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:30.137395 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:30.138135 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.139658 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.142179 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.143671 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.143808 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:30.144088 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:30.145677 (+ 1589us) server_negotiation.cc:299] Negotiation successful
0928 02:09:30.145832 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23724,"thread_start_us":23991,"threads_started":1}
I20260928 02:09:30.150226 21838 tablet_service.cc:1467] Processing CreateTablet for tablet 623e486ed0b64e328f6f8ffb51f131b1 (DEFAULT_TABLE table=test-table [id=71bbf5233513404f90014775674fca0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:09:30.151000 22099 tablet_service.cc:1467] Processing CreateTablet for tablet 623e486ed0b64e328f6f8ffb51f131b1 (DEFAULT_TABLE table=test-table [id=71bbf5233513404f90014775674fca0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:09:30.151160 21969 tablet_service.cc:1467] Processing CreateTablet for tablet 623e486ed0b64e328f6f8ffb51f131b1 (DEFAULT_TABLE table=test-table [id=71bbf5233513404f90014775674fca0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:09:30.151799 21838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 623e486ed0b64e328f6f8ffb51f131b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:30.152294 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 623e486ed0b64e328f6f8ffb51f131b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:30.152433 22099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 623e486ed0b64e328f6f8ffb51f131b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:30.163822 22182 tablet_bootstrap.cc:492] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63: Bootstrap starting.
I20260928 02:09:30.166478 22183 tablet_bootstrap.cc:492] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6: Bootstrap starting.
I20260928 02:09:30.168032 22184 tablet_bootstrap.cc:492] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8: Bootstrap starting.
I20260928 02:09:30.168207 22182 tablet_bootstrap.cc:654] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:30.169567 22182 log.cc:824] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:30.170364 22183 tablet_bootstrap.cc:654] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:30.170912 21903 heartbeater.cc:503] Master 127.1.182.190:36705 was elected leader, sending a full tablet report...
I20260928 02:09:30.171608 22184 tablet_bootstrap.cc:654] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:30.171928 22183 log.cc:824] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:30.172874 22184 log.cc:824] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:30.175371 22182 tablet_bootstrap.cc:492] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63: No bootstrap required, opened a new log
I20260928 02:09:30.175781 22182 ts_tablet_manager.cc:1397] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260928 02:09:30.181169 22183 tablet_bootstrap.cc:492] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6: No bootstrap required, opened a new log
I20260928 02:09:30.181437 22183 ts_tablet_manager.cc:1397] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260928 02:09:30.184723 22184 tablet_bootstrap.cc:492] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8: No bootstrap required, opened a new log
I20260928 02:09:30.184993 22184 ts_tablet_manager.cc:1397] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.006s
I20260928 02:09:30.187431 22182 raft_consensus.cc:348] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241e13727f5494c975691539f2def63" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 35793 } } peers { permanent_uuid: "1ea270e37cce4d5883caeded1accf4e8" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 36185 } } peers { permanent_uuid: "98bf90f55c734da6a8573ead733403b6" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 45893 } }
I20260928 02:09:30.187922 22182 raft_consensus.cc:374] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:30.188055 22182 raft_consensus.cc:729] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5241e13727f5494c975691539f2def63, State: Initialized, Role: FOLLOWER
I20260928 02:09:30.188761 22182 consensus_queue.cc:260] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241e13727f5494c975691539f2def63" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 35793 } } peers { permanent_uuid: "1ea270e37cce4d5883caeded1accf4e8" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 36185 } } peers { permanent_uuid: "98bf90f55c734da6a8573ead733403b6" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 45893 } }
I20260928 02:09:30.189909 22034 heartbeater.cc:503] Master 127.1.182.190:36705 was elected leader, sending a full tablet report...
I20260928 02:09:30.190707 22182 ts_tablet_manager.cc:1428] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260928 02:09:30.191882 22183 raft_consensus.cc:348] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241e13727f5494c975691539f2def63" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 35793 } } peers { permanent_uuid: "1ea270e37cce4d5883caeded1accf4e8" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 36185 } } peers { permanent_uuid: "98bf90f55c734da6a8573ead733403b6" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 45893 } }
I20260928 02:09:30.192322 22183 raft_consensus.cc:374] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:30.192425 22183 raft_consensus.cc:729] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98bf90f55c734da6a8573ead733403b6, State: Initialized, Role: FOLLOWER
I20260928 02:09:30.193091 22183 consensus_queue.cc:260] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241e13727f5494c975691539f2def63" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 35793 } } peers { permanent_uuid: "1ea270e37cce4d5883caeded1accf4e8" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 36185 } } peers { permanent_uuid: "98bf90f55c734da6a8573ead733403b6" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 45893 } }
I20260928 02:09:30.193588 22184 raft_consensus.cc:348] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241e13727f5494c975691539f2def63" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 35793 } } peers { permanent_uuid: "1ea270e37cce4d5883caeded1accf4e8" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 36185 } } peers { permanent_uuid: "98bf90f55c734da6a8573ead733403b6" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 45893 } }
I20260928 02:09:30.194041 22184 raft_consensus.cc:374] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:30.194169 22184 raft_consensus.cc:729] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ea270e37cce4d5883caeded1accf4e8, State: Initialized, Role: FOLLOWER
I20260928 02:09:30.194770 22184 consensus_queue.cc:260] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241e13727f5494c975691539f2def63" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 35793 } } peers { permanent_uuid: "1ea270e37cce4d5883caeded1accf4e8" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 36185 } } peers { permanent_uuid: "98bf90f55c734da6a8573ead733403b6" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 45893 } }
I20260928 02:09:30.194993 22183 ts_tablet_manager.cc:1428] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260928 02:09:30.195935 22164 heartbeater.cc:503] Master 127.1.182.190:36705 was elected leader, sending a full tablet report...
I20260928 02:09:30.196489 22184 ts_tablet_manager.cc:1428] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
W20260928 02:09:30.304039 22165 tablet.cc:2369] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:09:30.345502 22035 tablet.cc:2369] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:09:30.395639 21904 tablet.cc:2369] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:09:30.407052 22190 raft_consensus.cc:482] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:09:30.407392 22190 raft_consensus.cc:504] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241e13727f5494c975691539f2def63" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 35793 } } peers { permanent_uuid: "1ea270e37cce4d5883caeded1accf4e8" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 36185 } } peers { permanent_uuid: "98bf90f55c734da6a8573ead733403b6" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 45893 } }
I20260928 02:09:30.408974 22190 leader_election.cc:288] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5241e13727f5494c975691539f2def63 (127.1.182.130:35793), 1ea270e37cce4d5883caeded1accf4e8 (127.1.182.131:36185)
I20260928 02:09:30.418490 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.409411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:60647 (local address 127.1.182.131:36185)
0928 02:09:30.409604 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:30.409613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:30.410399 (+ 786us) server_negotiation.cc:407] Connection header received
0928 02:09:30.410641 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:30.410646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:30.410731 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:30.410824 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:30.411717 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.412917 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.416129 (+ 3212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.417373 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.417557 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:30.417632 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:30.418103 (+ 471us) server_negotiation.cc:299] Negotiation successful
0928 02:09:30.418230 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:09:30.418490 22177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.409549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.182.129:46835 (local address 127.1.182.130:35793)
0928 02:09:30.409747 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:30.409754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:30.410486 (+ 732us) server_negotiation.cc:407] Connection header received
0928 02:09:30.410825 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:30.410835 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:30.410984 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:30.411115 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:09:30.411891 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.412915 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.416130 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.417373 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.417557 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:30.417632 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:30.418103 (+ 471us) server_negotiation.cc:299] Negotiation successful
0928 02:09:30.418232 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:09:30.418730 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.409412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.130:35793 (local address 127.1.182.129:46835)
0928 02:09:30.410392 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:30.410408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:30.410584 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:30.411130 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:30.411138 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:30.411296 (+ 158us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:30.411739 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:30.411754 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.413052 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.413074 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:30.415856 (+ 2782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:30.415872 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.417613 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.417619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:30.417853 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:30.417961 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 02:09:30.418066 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:09:30.418260 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":220,"spinlock_wait_cycles":27776,"thread_start_us":365,"threads_started":1}
I20260928 02:09:30.418768 22193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.409320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.131:36185 (local address 127.1.182.129:60647)
0928 02:09:30.410221 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:30.410271 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:09:30.410439 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:09:30.411045 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:09:30.411050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:09:30.411110 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:09:30.411543 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:30.411569 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.413050 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.413055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:30.415859 (+ 2804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:09:30.415872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.417590 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.417602 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:09:30.417848 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:30.417960 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 02:09:30.418057 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 02:09:30.418261 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":300,"spinlock_wait_cycles":23296,"thread_start_us":1132,"threads_started":1}
I20260928 02:09:30.419997 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "623e486ed0b64e328f6f8ffb51f131b1" candidate_uuid: "98bf90f55c734da6a8573ead733403b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ea270e37cce4d5883caeded1accf4e8" is_pre_election: true
I20260928 02:09:30.419996 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "623e486ed0b64e328f6f8ffb51f131b1" candidate_uuid: "98bf90f55c734da6a8573ead733403b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5241e13727f5494c975691539f2def63" is_pre_election: true
I20260928 02:09:30.420615 21989 raft_consensus.cc:2454] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98bf90f55c734da6a8573ead733403b6 in term 0.
I20260928 02:09:30.420615 22119 raft_consensus.cc:2454] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98bf90f55c734da6a8573ead733403b6 in term 0.
I20260928 02:09:30.421295 21793 leader_election.cc:302] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5241e13727f5494c975691539f2def63, 98bf90f55c734da6a8573ead733403b6; no voters:
I20260928 02:09:30.421696 22190 raft_consensus.cc:2781] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:09:30.421916 22190 raft_consensus.cc:482] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:09:30.422036 22190 raft_consensus.cc:3037] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:30.424314 22190 raft_consensus.cc:504] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241e13727f5494c975691539f2def63" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 35793 } } peers { permanent_uuid: "1ea270e37cce4d5883caeded1accf4e8" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 36185 } } peers { permanent_uuid: "98bf90f55c734da6a8573ead733403b6" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 45893 } }
I20260928 02:09:30.424924 22190 leader_election.cc:288] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 [CANDIDATE]: Term 1 election: Requested vote from peers 5241e13727f5494c975691539f2def63 (127.1.182.130:35793), 1ea270e37cce4d5883caeded1accf4e8 (127.1.182.131:36185)
I20260928 02:09:30.425405 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "623e486ed0b64e328f6f8ffb51f131b1" candidate_uuid: "98bf90f55c734da6a8573ead733403b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ea270e37cce4d5883caeded1accf4e8"
I20260928 02:09:30.425392 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "623e486ed0b64e328f6f8ffb51f131b1" candidate_uuid: "98bf90f55c734da6a8573ead733403b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5241e13727f5494c975691539f2def63"
I20260928 02:09:30.425599 22119 raft_consensus.cc:3037] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:30.425655 21989 raft_consensus.cc:3037] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:30.427842 22119 raft_consensus.cc:2454] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98bf90f55c734da6a8573ead733403b6 in term 1.
I20260928 02:09:30.427992 21989 raft_consensus.cc:2454] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98bf90f55c734da6a8573ead733403b6 in term 1.
I20260928 02:09:30.428292 21794 leader_election.cc:302] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ea270e37cce4d5883caeded1accf4e8, 98bf90f55c734da6a8573ead733403b6; no voters:
I20260928 02:09:30.428603 22190 raft_consensus.cc:2781] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:30.429032 22190 raft_consensus.cc:686] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 [term 1 LEADER]: Becoming Leader. State: Replica: 98bf90f55c734da6a8573ead733403b6, State: Running, Role: LEADER
I20260928 02:09:30.429504 22190 consensus_queue.cc:237] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241e13727f5494c975691539f2def63" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 35793 } } peers { permanent_uuid: "1ea270e37cce4d5883caeded1accf4e8" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 36185 } } peers { permanent_uuid: "98bf90f55c734da6a8573ead733403b6" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 45893 } }
I20260928 02:09:30.434384 21715 catalog_manager.cc:5495] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98bf90f55c734da6a8573ead733403b6 (127.1.182.129). New cstate: current_term: 1 leader_uuid: "98bf90f55c734da6a8573ead733403b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5241e13727f5494c975691539f2def63" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 35793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ea270e37cce4d5883caeded1accf4e8" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 36185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98bf90f55c734da6a8573ead733403b6" member_type: VOTER last_known_addr { host: "127.1.182.129" port: 45893 } health_report { overall_health: HEALTHY } } }
I20260928 02:09:30.486562 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.480001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52348 (local address 127.1.182.129:45893)
0928 02:09:30.480245 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:30.480250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:30.480268 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:09:30.480596 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:30.480604 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:30.480714 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:30.480877 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:09:30.481458 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.482746 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.484467 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.484904 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.485001 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:30.485451 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:09:30.485990 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:09:30.486102 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:30.486172 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 02:09:30.486234 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260928 02:09:30.494805 21989 raft_consensus.cc:1261] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63 [term 1 FOLLOWER]: Refusing update from remote peer 98bf90f55c734da6a8573ead733403b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:09:30.494804 22119 raft_consensus.cc:1261] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8 [term 1 FOLLOWER]: Refusing update from remote peer 98bf90f55c734da6a8573ead733403b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:09:30.495857 22194 consensus_queue.cc:1035] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5241e13727f5494c975691539f2def63" member_type: VOTER last_known_addr { host: "127.1.182.130" port: 35793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:09:30.496205 22190 consensus_queue.cc:1035] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ea270e37cce4d5883caeded1accf4e8" member_type: VOTER last_known_addr { host: "127.1.182.131" port: 36185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:09:30.507239 22197 mvcc.cc:205] Tried to move back new op lower bound from 7334139373536497664 to 7334139373282910208. Current Snapshot: MvccSnapshot[applied={T|T < 7334139373536497664}]
I20260928 02:09:30.520738 21714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:09:30.521072 21714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:09:30.525053 21714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bb488d8254d4917a27463727237c1a6: Sending DeleteTablet for 3 replicas of tablet 623e486ed0b64e328f6f8ffb51f131b1
I20260928 02:09:30.526261 21969 tablet_service.cc:1514] Processing DeleteTablet for tablet 623e486ed0b64e328f6f8ffb51f131b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:09:30 UTC) from {username='slave'} at 127.0.0.1:53830
I20260928 02:09:30.526332 22099 tablet_service.cc:1514] Processing DeleteTablet for tablet 623e486ed0b64e328f6f8ffb51f131b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:09:30 UTC) from {username='slave'} at 127.0.0.1:41180
I20260928 02:09:30.526607 21838 tablet_service.cc:1514] Processing DeleteTablet for tablet 623e486ed0b64e328f6f8ffb51f131b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:09:30 UTC) from {username='slave'} at 127.0.0.1:52334
I20260928 02:09:30.531525 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 21776
I20260928 02:09:30.538599 22208 tablet_replica.cc:330] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6: stopping tablet replica
I20260928 02:09:30.539179 22208 raft_consensus.cc:2229] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:09:30.540031 22208 raft_consensus.cc:2258] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:09:30.545527 22207 tablet_replica.cc:330] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8: stopping tablet replica
I20260928 02:09:30.546092 22207 raft_consensus.cc:2229] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:09:30.546633 22207 raft_consensus.cc:2258] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:09:30.547703 22206 tablet_replica.cc:330] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63: stopping tablet replica
I20260928 02:09:30.548262 22206 raft_consensus.cc:2229] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:09:30.548849 22206 raft_consensus.cc:2258] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:09:30.548890 22207 ts_tablet_manager.cc:1905] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:09:30.549688 22208 ts_tablet_manager.cc:1905] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:09:30.551149 22206 ts_tablet_manager.cc:1905] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:09:30.555977 22207 ts_tablet_manager.cc:1918] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:09:30.556154 22207 log.cc:1196] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-2/wal/wals/623e486ed0b64e328f6f8ffb51f131b1
I20260928 02:09:30.556661 22207 ts_tablet_manager.cc:1939] T 623e486ed0b64e328f6f8ffb51f131b1 P 1ea270e37cce4d5883caeded1accf4e8: Deleting consensus metadata
I20260928 02:09:30.558496 21702 catalog_manager.cc:4841] TS 1ea270e37cce4d5883caeded1accf4e8 (127.1.182.131:36185): tablet 623e486ed0b64e328f6f8ffb51f131b1 (table test-table [id=71bbf5233513404f90014775674fca0f]) successfully deleted
I20260928 02:09:30.558606 22206 ts_tablet_manager.cc:1918] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:09:30.558763 22206 log.cc:1196] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-1/wal/wals/623e486ed0b64e328f6f8ffb51f131b1
I20260928 02:09:30.559298 22206 ts_tablet_manager.cc:1939] T 623e486ed0b64e328f6f8ffb51f131b1 P 5241e13727f5494c975691539f2def63: Deleting consensus metadata
I20260928 02:09:30.560707 21701 catalog_manager.cc:4841] TS 5241e13727f5494c975691539f2def63 (127.1.182.130:35793): tablet 623e486ed0b64e328f6f8ffb51f131b1 (table test-table [id=71bbf5233513404f90014775674fca0f]) successfully deleted
I20260928 02:09:30.563030 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.532452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52360 (local address 127.1.182.129:45893)
0928 02:09:30.532643 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:30.532651 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:30.546788 (+ 14137us) server_negotiation.cc:407] Connection header received
0928 02:09:30.547381 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:30.547388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:30.549275 (+ 1887us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:30.549370 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:30.550087 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.551426 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.553416 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.554091 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.554200 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:30.561484 (+ 7284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:30.561496 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:30.561510 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:30.562172 (+ 662us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:30.562219 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:30.562239 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:30.562391 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:30.562528 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 02:09:30.562638 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:09:30.563395 22208 ts_tablet_manager.cc:1918] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:09:30.563560 22208 log.cc:1196] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6: Deleting WAL directory at /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561052730740-1754-0/minicluster-data/ts-0/wal/wals/623e486ed0b64e328f6f8ffb51f131b1
I20260928 02:09:30.563764 21898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:30.674561 22208 ts_tablet_manager.cc:1939] T 623e486ed0b64e328f6f8ffb51f131b1 P 98bf90f55c734da6a8573ead733403b6: Deleting consensus metadata
I20260928 02:09:30.675159 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 21776
W20260928 02:09:30.693450 21700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.182.129:45893 (error 108)
I20260928 02:09:30.694265 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 21907
I20260928 02:09:30.695533 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.694665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:45893 (local address 127.0.0.1:52376)
0928 02:09:30.695159 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:30.695293 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:45893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":248,"threads_started":1}
W20260928 02:09:30.695812 21700 catalog_manager.cc:4584] TS 98bf90f55c734da6a8573ead733403b6 (127.1.182.129:45893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 623e486ed0b64e328f6f8ffb51f131b1: Network error: Client connection negotiation failed: client connection to 127.1.182.129:45893: connect: Connection refused (error 111)
I20260928 02:09:30.701608 22177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.695008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.1.182.130:35793)
0928 02:09:30.695263 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:30.695269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:30.695290 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:09:30.695475 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:30.695481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:30.695624 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:30.695732 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:30.696352 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.697316 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.698533 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.698935 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.699030 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:30.700250 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:30.700258 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:30.700272 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:30.700844 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:30.700888 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:30.700908 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:30.700995 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:30.701161 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 02:09:30.701248 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260928 02:09:30.702297 22029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:30.749368 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.748910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:45893 (local address 127.0.0.1:52388)
0928 02:09:30.749121 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:30.749219 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:45893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 02:09:30.750538 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.750089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:45893 (local address 127.0.0.1:52398)
0928 02:09:30.750300 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:30.750385 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:45893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 02:09:30.791539 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 21907
I20260928 02:09:30.809478 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 22037
I20260928 02:09:30.810889 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.810482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:45893 (local address 127.0.0.1:52404)
0928 02:09:30.810667 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:30.810751 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:45893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 02:09:30.812067 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.811702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:45893 (local address 127.0.0.1:52406)
0928 02:09:30.811842 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:30.811912 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:45893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 02:09:30.816893 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.810268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.1.182.131:36185)
0928 02:09:30.810456 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:30.810464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:30.810550 (+ 86us) server_negotiation.cc:407] Connection header received
0928 02:09:30.810769 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:30.810775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:30.810870 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:30.810997 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:30.811580 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.812518 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.813757 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:30.814132 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:30.814226 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:30.815457 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:30.815471 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:30.815497 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:30.816178 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:30.816226 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:30.816246 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:30.816342 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:30.816442 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 02:09:30.816545 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:09:30.817516 22159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:30.893039 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.892467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:45893 (local address 127.0.0.1:52414)
0928 02:09:30.892733 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:30.892850 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:45893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 02:09:30.894255 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:30.893787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.129:45893 (local address 127.0.0.1:52418)
0928 02:09:30.894008 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 02:09:30.894100 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.129:45893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260928 02:09:30.906591 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 22037
I20260928 02:09:30.922976 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 21684
I20260928 02:09:30.924171 21746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:31.034971 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 21684
2026-09-28T02:09:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2951 ms)
[----------] 6 tests from EncryptionPolicyTest (22372 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:09:31 dist-test-slave-23bp krb5kdc[22218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:09:31 dist-test-slave-23bp krb5kdc[22218](info): set up 2 sockets
Sep 28 02:09:31 dist-test-slave-23bp krb5kdc[22218](info): commencing operation
krb5kdc: starting...
W20260928 02:09:32.149837 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:09:32 dist-test-slave-23bp krb5kdc[22218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:09:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:09:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:09:32.403688 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32785
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:32.734694 22231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:32.735073 22231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:32.735266 22231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:32.742412 22231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:09:32.742583 22231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:32.742697 22231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:32.742806 22231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:09:32.742897 22231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:09:32.753019 22231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32785
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:32.754366 22231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:32.755733 22231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:32.767627 22239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:32.767719 22237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:32.767699 22236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:32.768507 22231 server_base.cc:1034] running on GCE node
I20260928 02:09:32.769302 22231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:32.770715 22231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:32.771960 22231 hybrid_clock.cc:648] HybridClock initialized: now 1790561372771877 us; error 45 us; skew 500 ppm
I20260928 02:09:32.772344 22231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:32 dist-test-slave-23bp krb5kdc[22218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:32.775532 22231 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:09:32.777063 22231 webserver.cc:460] Webserver started at http://127.0.0.1:43261/ using document root <none> and password file <none>
I20260928 02:09:32.777776 22231 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:32.777974 22231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:32.778307 22231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:32.781378 22231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "19da1e6b96f843df88eb31548874b168"
format_stamp: "Formatted at 2026-09-28 02:09:32 on dist-test-slave-23bp"
server_key: "62c69dec104c71149f5504afe3a1b865"
server_key_iv: "0b66409cc29b498ade4dc709bcda704e"
server_key_version: "encryptionkey@0"
I20260928 02:09:32.782141 22231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "19da1e6b96f843df88eb31548874b168"
format_stamp: "Formatted at 2026-09-28 02:09:32 on dist-test-slave-23bp"
server_key: "62c69dec104c71149f5504afe3a1b865"
server_key_iv: "0b66409cc29b498ade4dc709bcda704e"
server_key_version: "encryptionkey@0"
I20260928 02:09:32.787487 22231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:09:32.790879 22246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:32.792351 22231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:09:32.792544 22231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "19da1e6b96f843df88eb31548874b168"
format_stamp: "Formatted at 2026-09-28 02:09:32 on dist-test-slave-23bp"
server_key: "62c69dec104c71149f5504afe3a1b865"
server_key_iv: "0b66409cc29b498ade4dc709bcda704e"
server_key_version: "encryptionkey@0"
I20260928 02:09:32.792747 22231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:32.809935 22231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:32.814316 22231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:32.814622 22231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:32.833799 22231 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32785
I20260928 02:09:32.833822 22298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32785 every 8 connection(s)
I20260928 02:09:32.835366 22231 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:09:32.837399 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 22231
I20260928 02:09:32.837618 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:09:32.838430 1754 external_mini_cluster.cc:1428] Setting key 48ecb7c63a665b3eb57f2e85c98b924f
I20260928 02:09:32.840010 22299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:32.852526 22299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168: Bootstrap starting.
Sep 28 02:09:32 dist-test-slave-23bp krb5kdc[22218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:09:32.856392 22299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:32.857528 22299 log.cc:824] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:32.860816 22299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168: No bootstrap required, opened a new log
I20260928 02:09:32.862087 22302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:32.840494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35749 (local address 127.0.0.1:32785)
0928 02:09:32.841748 (+ 1254us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:32.841760 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:32.841800 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:09:32.843569 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:32.843590 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:32.844076 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:32.844478 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:32.846051 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:32.847533 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:32.848791 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:32.849498 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:32.849714 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:32.854452 (+ 4738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:32.854460 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:32.854475 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:32.854535 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:32.857815 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:32.858743 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:32.858753 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:32.858764 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:32.858936 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:32.859518 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:32.859522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:32.859524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:32.860045 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:32.860276 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:32.860804 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 02:09:32.861121 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1050,"thread_start_us":833,"threads_started":1}
I20260928 02:09:32.866145 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:09:32.869002 22299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19da1e6b96f843df88eb31548874b168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32785 } }
I20260928 02:09:32.869365 22299 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:32.869483 22299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19da1e6b96f843df88eb31548874b168, State: Initialized, Role: FOLLOWER
I20260928 02:09:32.870060 22299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19da1e6b96f843df88eb31548874b168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32785 } }
I20260928 02:09:32.870280 22299 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:09:32.870407 22299 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:09:32.870589 22299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:32.872815 22299 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19da1e6b96f843df88eb31548874b168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32785 } }
I20260928 02:09:32.873306 22299 leader_election.cc:302] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19da1e6b96f843df88eb31548874b168; no voters:
I20260928 02:09:32.873958 22299 leader_election.cc:288] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:09:32.874122 22306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:32.875490 22306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168 [term 1 LEADER]: Becoming Leader. State: Replica: 19da1e6b96f843df88eb31548874b168, State: Running, Role: LEADER
I20260928 02:09:32.876088 22306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19da1e6b96f843df88eb31548874b168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32785 } }
I20260928 02:09:32.876863 22299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:09:32.880606 22309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19da1e6b96f843df88eb31548874b168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19da1e6b96f843df88eb31548874b168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32785 } } }
I20260928 02:09:32.881063 22309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:32.881672 22310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19da1e6b96f843df88eb31548874b168. Latest consensus state: current_term: 1 leader_uuid: "19da1e6b96f843df88eb31548874b168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19da1e6b96f843df88eb31548874b168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32785 } } }
I20260928 02:09:32.882061 22310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:32.897560 22302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:32.876467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44353 (local address 127.0.0.1:32785)
0928 02:09:32.876659 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:32.876665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:32.876693 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:09:32.876862 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:32.876867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:32.876964 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:32.877070 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:32.878797 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:32.880012 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:32.881833 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:32.882758 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:32.882911 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:32.887117 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:32.887127 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:32.887136 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:32.887188 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:32.891794 (+ 4606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:32.892978 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:32.892988 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:32.892994 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:32.893313 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:32.894044 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:32.894052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:32.894058 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:32.894559 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:32.894950 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:32.896641 (+ 1691us) server_negotiation.cc:299] Negotiation successful
0928 02:09:32.897029 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:09:32.903975 22316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:09:32.910095 22316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:09:32.919842 22316 catalog_manager.cc:1347] Generated new cluster ID: ad7fe2152cfe48a7ac9ed131a3d201b3
I20260928 02:09:32.920028 22316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:09:32.929291 22316 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:09:32.931125 22316 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:09:32.941090 22316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19da1e6b96f843df88eb31548874b168: Generated new TSK 0
I20260928 02:09:32.942153 22316 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:09:32.950574 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 22231
I20260928 02:09:32.952683 22294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:33.042872 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 22231
W20260928 02:09:33.060431 22216 connection.cc:535] client connection to 127.0.0.1:32785 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32785): Connection reset by peer (error 104)
2026-09-28T02:09:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2001 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:09:33 dist-test-slave-23bp krb5kdc[22332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:09:33 dist-test-slave-23bp krb5kdc[22332](info): set up 2 sockets
Sep 28 02:09:33 dist-test-slave-23bp krb5kdc[22332](info): commencing operation
krb5kdc: starting...
W20260928 02:09:34.165757 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:09:34 dist-test-slave-23bp krb5kdc[22332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:09:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:09:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:09:34.425911 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46177
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:34.764559 22345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:34.764854 22345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:34.764946 22345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:34.772367 22345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:09:34.772486 22345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:34.772532 22345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:34.772593 22345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:09:34.772639 22345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:09:34.782660 22345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46177
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:34.784049 22345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:34.785506 22345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:34.797761 22352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:34.797994 22354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:34.797734 22351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:34.798647 22345 server_base.cc:1034] running on GCE node
I20260928 02:09:34.799551 22345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:34.801074 22345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:34.802307 22345 hybrid_clock.cc:648] HybridClock initialized: now 1790561374802277 us; error 63 us; skew 500 ppm
I20260928 02:09:34.802659 22345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:34 dist-test-slave-23bp krb5kdc[22332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:34.805786 22345 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:09:34.807381 22345 webserver.cc:460] Webserver started at http://127.0.0.1:36365/ using document root <none> and password file <none>
I20260928 02:09:34.808079 22345 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:34.808228 22345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:34.808537 22345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:34.811638 22345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "65ccd9eb761444a0b906684bbbbafd8a"
format_stamp: "Formatted at 2026-09-28 02:09:34 on dist-test-slave-23bp"
server_key: "5f8721a1eed70779ab6c5eb326f78784"
server_key_iv: "b6d0081d855196973b3686d242a67705"
server_key_version: "encryptionkey@0"
I20260928 02:09:34.812381 22345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "65ccd9eb761444a0b906684bbbbafd8a"
format_stamp: "Formatted at 2026-09-28 02:09:34 on dist-test-slave-23bp"
server_key: "5f8721a1eed70779ab6c5eb326f78784"
server_key_iv: "b6d0081d855196973b3686d242a67705"
server_key_version: "encryptionkey@0"
I20260928 02:09:34.817478 22345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:09:34.820824 22361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:34.822165 22345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:09:34.822345 22345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "65ccd9eb761444a0b906684bbbbafd8a"
format_stamp: "Formatted at 2026-09-28 02:09:34 on dist-test-slave-23bp"
server_key: "5f8721a1eed70779ab6c5eb326f78784"
server_key_iv: "b6d0081d855196973b3686d242a67705"
server_key_version: "encryptionkey@0"
I20260928 02:09:34.822540 22345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:34.844532 22345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:34.849293 22345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:34.849591 22345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:34.868531 22345 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46177
I20260928 02:09:34.868571 22412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46177 every 8 connection(s)
I20260928 02:09:34.869915 22345 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:09:34.872279 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 22345
I20260928 02:09:34.872466 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:09:34.872995 1754 external_mini_cluster.cc:1428] Setting key 75ad0b8bc4fd2d53814674990cddadae
I20260928 02:09:34.874370 22413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:34.886289 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:34.874847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43281 (local address 127.0.0.1:46177)
0928 02:09:34.875703 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:34.875716 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:34.875757 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:09:34.877394 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:34.877423 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:34.877908 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:34.878319 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:34.878861 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:34.880273 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:34.881590 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:34.882243 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:34.882451 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:34.883951 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:34.883959 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:34.883973 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:34.884602 (+ 629us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:34.884669 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:34.884695 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:34.884869 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:34.885148 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 02:09:34.885317 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":374,"threads_started":1}
I20260928 02:09:34.886385 22413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a: Bootstrap starting.
I20260928 02:09:34.890059 22413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:34.890441 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:09:34.891209 22413 log.cc:824] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:34.894137 22413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a: No bootstrap required, opened a new log
I20260928 02:09:34.901098 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:34.894872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39335 (local address 127.0.0.1:46177)
0928 02:09:34.895130 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:34.895137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:34.895434 (+ 297us) server_negotiation.cc:407] Connection header received
0928 02:09:34.895654 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:34.895658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:34.895718 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:34.895838 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:34.896331 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:34.897145 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:34.898482 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:34.898962 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:34.899159 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:34.900327 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:34.900331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:34.900335 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:34.900354 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:34.900409 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:34.900427 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:34.900586 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:34.900768 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 02:09:34.900822 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:09:34.902936 22413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ccd9eb761444a0b906684bbbbafd8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46177 } }
I20260928 02:09:34.903388 22413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:34.903508 22413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ccd9eb761444a0b906684bbbbafd8a, State: Initialized, Role: FOLLOWER
I20260928 02:09:34.904088 22413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ccd9eb761444a0b906684bbbbafd8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46177 } }
I20260928 02:09:34.904309 22413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:09:34.904464 22413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:09:34.904652 22413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:34.906912 22413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ccd9eb761444a0b906684bbbbafd8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46177 } }
I20260928 02:09:34.907497 22413 leader_election.cc:302] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65ccd9eb761444a0b906684bbbbafd8a; no voters:
I20260928 02:09:34.908088 22413 leader_election.cc:288] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:09:34.908242 22423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:34.909014 22423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a [term 1 LEADER]: Becoming Leader. State: Replica: 65ccd9eb761444a0b906684bbbbafd8a, State: Running, Role: LEADER
I20260928 02:09:34.909583 22423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ccd9eb761444a0b906684bbbbafd8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46177 } }
I20260928 02:09:34.910322 22413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:09:34.912640 22425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65ccd9eb761444a0b906684bbbbafd8a. Latest consensus state: current_term: 1 leader_uuid: "65ccd9eb761444a0b906684bbbbafd8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ccd9eb761444a0b906684bbbbafd8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46177 } } }
I20260928 02:09:34.913039 22425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:34.914017 22428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:09:34.913894 22424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65ccd9eb761444a0b906684bbbbafd8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ccd9eb761444a0b906684bbbbafd8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46177 } } }
I20260928 02:09:34.915024 22424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:34.919746 22428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:09:34.929312 22428 catalog_manager.cc:1347] Generated new cluster ID: 3f79bb8e855e467089fd698823837bb0
I20260928 02:09:34.929445 22428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:09:34.955217 22428 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:09:34.957145 22428 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:09:34.978865 22428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65ccd9eb761444a0b906684bbbbafd8a: Generated new TSK 0
I20260928 02:09:34.979986 22428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:09:34.989702 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 22345
W20260928 02:09:34.990885 22408 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43281
W20260928 02:09:34.991729 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 22345: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:09:34.991925 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 22345
W20260928 02:09:35.006116 22330 connection.cc:535] client connection to 127.0.0.1:46177 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46177): Connection reset by peer (error 104)
2026-09-28T02:09:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1945 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:09:35 dist-test-slave-23bp krb5kdc[22444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:09:35 dist-test-slave-23bp krb5kdc[22444](info): set up 2 sockets
Sep 28 02:09:35 dist-test-slave-23bp krb5kdc[22444](info): commencing operation
krb5kdc: starting...
W20260928 02:09:36.105808 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:09:36 dist-test-slave-23bp krb5kdc[22444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:09:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:09:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:09:36.362517 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34287
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:36.700938 22457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:36.701246 22457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:36.701452 22457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:36.708899 22457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:09:36.709007 22457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:36.709142 22457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:36.709249 22457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:09:36.709327 22457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:09:36.719264 22457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34287
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:36.720556 22457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:36.721905 22457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:36.734131 22462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:36.734071 22463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:36.734092 22465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:36.735204 22457 server_base.cc:1034] running on GCE node
I20260928 02:09:36.736047 22457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:36.737452 22457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:36.738651 22457 hybrid_clock.cc:648] HybridClock initialized: now 1790561376738599 us; error 54 us; skew 500 ppm
I20260928 02:09:36.738987 22457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:36 dist-test-slave-23bp krb5kdc[22444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:36.746124 22457 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:09:36.747709 22457 webserver.cc:460] Webserver started at http://127.0.0.1:32769/ using document root <none> and password file <none>
I20260928 02:09:36.748417 22457 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:36.748567 22457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:36.748869 22457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:36.751904 22457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "b75b90b2a6304bd89baac9cd0f62b560"
format_stamp: "Formatted at 2026-09-28 02:09:36 on dist-test-slave-23bp"
server_key: "43345d0c789c752f3e2b357f14f3c386"
server_key_iv: "35da1ba35a949f2c9fc3e0b9626392ac"
server_key_version: "encryptionkey@0"
I20260928 02:09:36.752667 22457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "b75b90b2a6304bd89baac9cd0f62b560"
format_stamp: "Formatted at 2026-09-28 02:09:36 on dist-test-slave-23bp"
server_key: "43345d0c789c752f3e2b357f14f3c386"
server_key_iv: "35da1ba35a949f2c9fc3e0b9626392ac"
server_key_version: "encryptionkey@0"
I20260928 02:09:36.757934 22457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:09:36.761440 22472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:36.762712 22457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:09:36.762894 22457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "b75b90b2a6304bd89baac9cd0f62b560"
format_stamp: "Formatted at 2026-09-28 02:09:36 on dist-test-slave-23bp"
server_key: "43345d0c789c752f3e2b357f14f3c386"
server_key_iv: "35da1ba35a949f2c9fc3e0b9626392ac"
server_key_version: "encryptionkey@0"
I20260928 02:09:36.763130 22457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:36.768656 22457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:36.773051 22457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:36.773353 22457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:36.791694 22457 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34287
I20260928 02:09:36.791731 22524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34287 every 8 connection(s)
I20260928 02:09:36.793128 22457 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:09:36.796717 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 22457
I20260928 02:09:36.796918 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:09:36.796927 22525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:36.797369 1754 external_mini_cluster.cc:1428] Setting key 691e772652b65f0514011f553ed9e9ac
I20260928 02:09:36.805771 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:36.799198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55597 (local address 127.0.0.1:34287)
0928 02:09:36.800055 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:36.800071 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:36.800124 (+ 53us) server_negotiation.cc:407] Connection header received
0928 02:09:36.801895 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:36.801922 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:36.802365 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:36.802827 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:36.803854 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:36.803861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:36.803877 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:36.803893 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:36.803968 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:36.803998 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:36.804218 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:36.804354 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 02:09:36.804555 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":337,"threads_started":1}
I20260928 02:09:36.809453 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:09:36.811493 22525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560: Bootstrap starting.
I20260928 02:09:36.814889 22525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:36.816102 22525 log.cc:824] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:36.816346 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:36.813877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48675 (local address 127.0.0.1:34287)
0928 02:09:36.814058 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:36.814064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:36.814355 (+ 291us) server_negotiation.cc:407] Connection header received
0928 02:09:36.814479 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:36.814483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:36.814549 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:36.814654 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:36.815690 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:36.815694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:36.815699 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:36.815717 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:36.815774 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:36.815794 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:36.815922 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:36.815984 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 02:09:36.816037 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:09:36.819123 22525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560: No bootstrap required, opened a new log
I20260928 02:09:36.826503 22525 raft_consensus.cc:348] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75b90b2a6304bd89baac9cd0f62b560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34287 } }
I20260928 02:09:36.827003 22525 raft_consensus.cc:374] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:36.827153 22525 raft_consensus.cc:729] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75b90b2a6304bd89baac9cd0f62b560, State: Initialized, Role: FOLLOWER
I20260928 02:09:36.827761 22525 consensus_queue.cc:260] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75b90b2a6304bd89baac9cd0f62b560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34287 } }
I20260928 02:09:36.828001 22525 raft_consensus.cc:388] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:09:36.828162 22525 raft_consensus.cc:482] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:09:36.828332 22525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:36.830587 22525 raft_consensus.cc:504] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75b90b2a6304bd89baac9cd0f62b560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34287 } }
I20260928 02:09:36.831123 22525 leader_election.cc:302] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b75b90b2a6304bd89baac9cd0f62b560; no voters:
I20260928 02:09:36.831732 22525 leader_election.cc:288] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:09:36.831871 22535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:36.832556 22535 raft_consensus.cc:686] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560 [term 1 LEADER]: Becoming Leader. State: Replica: b75b90b2a6304bd89baac9cd0f62b560, State: Running, Role: LEADER
I20260928 02:09:36.833047 22535 consensus_queue.cc:237] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75b90b2a6304bd89baac9cd0f62b560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34287 } }
I20260928 02:09:36.833945 22525 sys_catalog.cc:564] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:09:36.836025 22537 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b75b90b2a6304bd89baac9cd0f62b560. Latest consensus state: current_term: 1 leader_uuid: "b75b90b2a6304bd89baac9cd0f62b560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75b90b2a6304bd89baac9cd0f62b560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34287 } } }
I20260928 02:09:36.836499 22537 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:36.836963 22536 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b75b90b2a6304bd89baac9cd0f62b560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75b90b2a6304bd89baac9cd0f62b560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34287 } } }
I20260928 02:09:36.837571 22536 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:36.838974 22542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:09:36.844076 22542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:09:36.853235 22542 catalog_manager.cc:1347] Generated new cluster ID: b8ecd6288cfb40e9a2bba6a9c2781c46
I20260928 02:09:36.853358 22542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:09:36.865204 22542 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:09:36.865633 22542 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:09:36.875896 22542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b75b90b2a6304bd89baac9cd0f62b560: Generated new TSK 0
I20260928 02:09:36.876931 22542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:09:36.888974 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 22457
W20260928 02:09:36.890278 22520 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55597
W20260928 02:09:36.891037 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 22457: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:09:36.891232 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 22457
2026-09-28T02:09:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1902 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:09:36 dist-test-slave-23bp krb5kdc[22556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:09:36 dist-test-slave-23bp krb5kdc[22556](info): set up 2 sockets
Sep 28 02:09:36 dist-test-slave-23bp krb5kdc[22556](info): commencing operation
krb5kdc: starting...
W20260928 02:09:38.001883 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:09:38 dist-test-slave-23bp krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:09:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:09:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:09:38.254521 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:38.583555 22569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:38.583936 22569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:38.584107 22569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:38.591501 22569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:09:38.591670 22569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:38.591790 22569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:38.591897 22569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:09:38.591982 22569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:09:38.602183 22569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:38.603591 22569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:38.604912 22569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:38.616978 22574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:38.617075 22575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:38.617055 22577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:38.618422 22569 server_base.cc:1034] running on GCE node
I20260928 02:09:38.619299 22569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:38.620713 22569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:38.621946 22569 hybrid_clock.cc:648] HybridClock initialized: now 1790561378621893 us; error 61 us; skew 500 ppm
I20260928 02:09:38.622354 22569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:38 dist-test-slave-23bp krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:38.625481 22569 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:09:38.626938 22569 webserver.cc:460] Webserver started at http://127.0.0.1:34943/ using document root <none> and password file <none>
I20260928 02:09:38.627722 22569 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:38.627864 22569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:38.628224 22569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:38.631207 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "b5a385b77f98456a9c6a6fa0965d5480"
format_stamp: "Formatted at 2026-09-28 02:09:38 on dist-test-slave-23bp"
server_key: "fe4d4cb52c0a49f0a20c2b1e7c2eaae1"
server_key_iv: "513db6c30a8b347fda8042c9e3657907"
server_key_version: "encryptionkey@0"
I20260928 02:09:38.632001 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "b5a385b77f98456a9c6a6fa0965d5480"
format_stamp: "Formatted at 2026-09-28 02:09:38 on dist-test-slave-23bp"
server_key: "fe4d4cb52c0a49f0a20c2b1e7c2eaae1"
server_key_iv: "513db6c30a8b347fda8042c9e3657907"
server_key_version: "encryptionkey@0"
I20260928 02:09:38.637058 22569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:09:38.640518 22584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:38.641852 22569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:09:38.642096 22569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "b5a385b77f98456a9c6a6fa0965d5480"
format_stamp: "Formatted at 2026-09-28 02:09:38 on dist-test-slave-23bp"
server_key: "fe4d4cb52c0a49f0a20c2b1e7c2eaae1"
server_key_iv: "513db6c30a8b347fda8042c9e3657907"
server_key_version: "encryptionkey@0"
I20260928 02:09:38.642297 22569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:38.662575 22569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:38.666769 22569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:38.667125 22569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:38.686425 22569 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37653
I20260928 02:09:38.686457 22635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37653 every 8 connection(s)
I20260928 02:09:38.687947 22569 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:09:38.688570 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 22569
I20260928 02:09:38.688778 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:09:38.689325 1754 external_mini_cluster.cc:1428] Setting key d467669f062063da88260134560480cb
I20260928 02:09:38.692806 22636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 02:09:38 dist-test-slave-23bp krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:09:38.705559 22636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480: Bootstrap starting.
I20260928 02:09:38.709167 22636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:38.710487 22636 log.cc:824] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:38.713124 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:38.691210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39719 (local address 127.0.0.1:37653)
0928 02:09:38.691995 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:38.692011 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:38.692059 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:09:38.694373 (+ 2314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:38.694401 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:38.694907 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:38.695346 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:38.696728 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:38.698766 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:38.700087 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:38.701110 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:38.701286 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:38.705804 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:38.705813 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:38.705828 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:38.705870 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:38.708906 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:38.709778 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:38.709790 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:38.709797 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:38.709951 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:38.710394 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:38.710397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:38.710399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:38.710980 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:38.711330 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:38.711763 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 02:09:38.712174 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":336,"threads_started":1}
I20260928 02:09:38.713454 22636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480: No bootstrap required, opened a new log
I20260928 02:09:38.717093 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:09:38.722177 22636 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a385b77f98456a9c6a6fa0965d5480" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } }
I20260928 02:09:38.722574 22636 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:38.722703 22636 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a385b77f98456a9c6a6fa0965d5480, State: Initialized, Role: FOLLOWER
I20260928 02:09:38.723328 22636 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a385b77f98456a9c6a6fa0965d5480" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } }
I20260928 02:09:38.723569 22636 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:09:38.723716 22636 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:09:38.723901 22636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:38.726152 22636 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a385b77f98456a9c6a6fa0965d5480" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } }
I20260928 02:09:38.726670 22636 leader_election.cc:302] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5a385b77f98456a9c6a6fa0965d5480; no voters:
I20260928 02:09:38.727371 22636 leader_election.cc:288] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:09:38.727463 22647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:38.728735 22647 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480 [term 1 LEADER]: Becoming Leader. State: Replica: b5a385b77f98456a9c6a6fa0965d5480, State: Running, Role: LEADER
I20260928 02:09:38.729319 22647 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a385b77f98456a9c6a6fa0965d5480" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } }
I20260928 02:09:38.729786 22636 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:09:38.732803 22648 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5a385b77f98456a9c6a6fa0965d5480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a385b77f98456a9c6a6fa0965d5480" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } } }
I20260928 02:09:38.733180 22648 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:38.735173 22649 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5a385b77f98456a9c6a6fa0965d5480. Latest consensus state: current_term: 1 leader_uuid: "b5a385b77f98456a9c6a6fa0965d5480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a385b77f98456a9c6a6fa0965d5480" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } } }
I20260928 02:09:38.735584 22649 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:38.740614 22655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:09:38.741935 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:38.721459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42409 (local address 127.0.0.1:37653)
0928 02:09:38.721651 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:38.721660 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:38.722061 (+ 401us) server_negotiation.cc:407] Connection header received
0928 02:09:38.722251 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:38.722256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:38.722324 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:38.722447 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:38.724152 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:38.724909 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:38.726178 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:38.726665 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:38.726794 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:38.729736 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:38.729741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:38.729744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:38.729776 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:38.734779 (+ 5003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:38.735959 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:38.735968 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:38.735974 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:38.739392 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:38.740008 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:38.740016 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:38.740022 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:38.740416 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:38.740731 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:38.741314 (+ 583us) server_negotiation.cc:299] Negotiation successful
0928 02:09:38.741543 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:09:38.746762 22655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:09:38.756134 22655 catalog_manager.cc:1347] Generated new cluster ID: 28f71e96cb324598b2a2a2d6245475db
I20260928 02:09:38.756235 22655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:09:38.777555 22655 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:09:38.779430 22655 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:09:38.788919 22655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5a385b77f98456a9c6a6fa0965d5480: Generated new TSK 0
I20260928 02:09:38.790033 22655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:09:38.800248 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 22569
I20260928 02:09:38.801734 22631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:38.895035 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 22569
2026-09-28T02:09:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2003 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:09:38 dist-test-slave-23bp krb5kdc[22669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:09:38 dist-test-slave-23bp krb5kdc[22669](info): set up 2 sockets
Sep 28 02:09:38 dist-test-slave-23bp krb5kdc[22669](info): commencing operation
krb5kdc: starting...
W20260928 02:09:40.065992 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:09:40 dist-test-slave-23bp krb5kdc[22669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:09:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:09:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:09:40.313822 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42893
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:40.648274 22682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:40.648619 22682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:40.648788 22682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:40.656229 22682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:09:40.656387 22682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:40.656497 22682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:40.656613 22682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:09:40.656697 22682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:09:40.666891 22682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:40.668269 22682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:40.669548 22682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:40.681651 22687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:40.681681 22690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:40.681615 22688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:40.682569 22682 server_base.cc:1034] running on GCE node
I20260928 02:09:40.683462 22682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:40.684862 22682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:40.686082 22682 hybrid_clock.cc:648] HybridClock initialized: now 1790561380686032 us; error 43 us; skew 500 ppm
I20260928 02:09:40.686460 22682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:40 dist-test-slave-23bp krb5kdc[22669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:40.689594 22682 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:09:40.690981 22682 webserver.cc:460] Webserver started at http://127.0.0.1:43373/ using document root <none> and password file <none>
I20260928 02:09:40.691695 22682 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:40.691840 22682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:40.692140 22682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:40.695048 22682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "2e4a8521ba694e10bec0fa4b63e50209"
format_stamp: "Formatted at 2026-09-28 02:09:40 on dist-test-slave-23bp"
server_key: "38d9116749ba14ce18c6186b03487b25"
server_key_iv: "a56159bb8c25c203b577aa0ee3fd7b96"
server_key_version: "encryptionkey@0"
I20260928 02:09:40.695892 22682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "2e4a8521ba694e10bec0fa4b63e50209"
format_stamp: "Formatted at 2026-09-28 02:09:40 on dist-test-slave-23bp"
server_key: "38d9116749ba14ce18c6186b03487b25"
server_key_iv: "a56159bb8c25c203b577aa0ee3fd7b96"
server_key_version: "encryptionkey@0"
I20260928 02:09:40.700891 22682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:09:40.704267 22697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:40.705582 22682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:09:40.705763 22682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "2e4a8521ba694e10bec0fa4b63e50209"
format_stamp: "Formatted at 2026-09-28 02:09:40 on dist-test-slave-23bp"
server_key: "38d9116749ba14ce18c6186b03487b25"
server_key_iv: "a56159bb8c25c203b577aa0ee3fd7b96"
server_key_version: "encryptionkey@0"
I20260928 02:09:40.705971 22682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:40.727898 22682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:40.731828 22682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:40.732120 22682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:40.750332 22682 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42893
I20260928 02:09:40.750365 22748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42893 every 8 connection(s)
I20260928 02:09:40.751729 22682 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:09:40.755470 22749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:40.758679 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 22682
I20260928 02:09:40.758860 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:09:40.759344 1754 external_mini_cluster.cc:1428] Setting key 12f33b4d63903ee432ec32412962510f
I20260928 02:09:40.767019 22749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209: Bootstrap starting.
I20260928 02:09:40.770226 22749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:40.771413 22749 log.cc:824] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:40.772396 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:40.761198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53863 (local address 127.0.0.1:42893)
0928 02:09:40.761884 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:40.761895 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:40.761943 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:09:40.763631 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:40.763690 (+ 59us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:40.764156 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:40.764604 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:40.765232 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:40.766507 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:40.767937 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:40.768621 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:40.768838 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:40.770019 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:40.770027 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:40.770040 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:40.770620 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:40.770701 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:40.770726 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:40.770853 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:40.771107 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 02:09:40.771372 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":292,"threads_started":1}
I20260928 02:09:40.774227 22749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209: No bootstrap required, opened a new log
I20260928 02:09:40.775964 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:09:40.784554 22749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4a8521ba694e10bec0fa4b63e50209" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42893 } }
I20260928 02:09:40.784937 22749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:40.785094 22749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4a8521ba694e10bec0fa4b63e50209, State: Initialized, Role: FOLLOWER
I20260928 02:09:40.785696 22749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4a8521ba694e10bec0fa4b63e50209" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42893 } }
I20260928 02:09:40.785953 22749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:09:40.786116 22749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:09:40.786288 22749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:40.787691 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:40.781141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33199 (local address 127.0.0.1:42893)
0928 02:09:40.781343 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:40.781349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:40.781746 (+ 397us) server_negotiation.cc:407] Connection header received
0928 02:09:40.781890 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:40.781895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:40.781974 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:40.782087 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:40.782534 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:40.783687 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:40.784979 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:40.785616 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:40.785757 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:40.786892 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:40.786899 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:40.786904 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:40.786924 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:40.786984 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:40.787008 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:40.787135 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:40.787253 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 02:09:40.787355 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:09:40.789144 22749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4a8521ba694e10bec0fa4b63e50209" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42893 } }
I20260928 02:09:40.789587 22749 leader_election.cc:302] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e4a8521ba694e10bec0fa4b63e50209; no voters:
I20260928 02:09:40.790086 22749 leader_election.cc:288] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:09:40.790225 22759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:40.790939 22759 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209 [term 1 LEADER]: Becoming Leader. State: Replica: 2e4a8521ba694e10bec0fa4b63e50209, State: Running, Role: LEADER
I20260928 02:09:40.791447 22759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4a8521ba694e10bec0fa4b63e50209" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42893 } }
I20260928 02:09:40.792363 22749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:09:40.794598 22760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e4a8521ba694e10bec0fa4b63e50209" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4a8521ba694e10bec0fa4b63e50209" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42893 } } }
I20260928 02:09:40.795598 22760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:40.795284 22761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e4a8521ba694e10bec0fa4b63e50209. Latest consensus state: current_term: 1 leader_uuid: "2e4a8521ba694e10bec0fa4b63e50209" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4a8521ba694e10bec0fa4b63e50209" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42893 } } }
I20260928 02:09:40.796064 22761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:40.799557 22769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:09:40.804819 22769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:09:40.813864 22769 catalog_manager.cc:1347] Generated new cluster ID: 25f21ca252f647ce99a85abfe48007da
I20260928 02:09:40.814020 22769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:09:40.846982 22769 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:09:40.848338 22769 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:09:40.856211 22769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e4a8521ba694e10bec0fa4b63e50209: Generated new TSK 0
I20260928 02:09:40.857030 22769 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:09:40.870776 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 22682
W20260928 02:09:40.871963 22744 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53863
W20260928 02:09:40.872720 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 22682: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:09:40.872891 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 22682
2026-09-28T02:09:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1974 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:09:40 dist-test-slave-23bp krb5kdc[22780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:09:40 dist-test-slave-23bp krb5kdc[22780](info): set up 2 sockets
Sep 28 02:09:40 dist-test-slave-23bp krb5kdc[22780](info): commencing operation
krb5kdc: starting...
W20260928 02:09:42.045948 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:09:42 dist-test-slave-23bp krb5kdc[22780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:09:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:09:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:09:42.310921 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43727
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:42.646469 22793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:42.646755 22793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:42.646986 22793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:42.654234 22793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:09:42.654398 22793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:42.654506 22793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:42.654618 22793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:09:42.654702 22793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:09:42.664628 22793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:42.665958 22793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:42.667423 22793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:42.679492 22801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:42.679581 22799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:42.679605 22798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:42.684633 22793 server_base.cc:1034] running on GCE node
I20260928 02:09:42.685521 22793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:42.686985 22793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:42.688251 22793 hybrid_clock.cc:648] HybridClock initialized: now 1790561382688205 us; error 31 us; skew 500 ppm
I20260928 02:09:42.688660 22793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:42 dist-test-slave-23bp krb5kdc[22780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:42.698230 22793 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:09:42.700048 22793 webserver.cc:460] Webserver started at http://127.0.0.1:41305/ using document root <none> and password file <none>
I20260928 02:09:42.700814 22793 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:42.700963 22793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:42.701277 22793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:42.704768 22793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "2ba82340d6aa4717a05002cd4e8f248a"
format_stamp: "Formatted at 2026-09-28 02:09:42 on dist-test-slave-23bp"
server_key: "b0e4afc4113b553d6433db34fea2d981"
server_key_iv: "c1db8e521f84e0e315c7ebcfa4068b3a"
server_key_version: "encryptionkey@0"
I20260928 02:09:42.705592 22793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "2ba82340d6aa4717a05002cd4e8f248a"
format_stamp: "Formatted at 2026-09-28 02:09:42 on dist-test-slave-23bp"
server_key: "b0e4afc4113b553d6433db34fea2d981"
server_key_iv: "c1db8e521f84e0e315c7ebcfa4068b3a"
server_key_version: "encryptionkey@0"
I20260928 02:09:42.711026 22793 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:09:42.714599 22808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:42.716034 22793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:09:42.716174 22793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "2ba82340d6aa4717a05002cd4e8f248a"
format_stamp: "Formatted at 2026-09-28 02:09:42 on dist-test-slave-23bp"
server_key: "b0e4afc4113b553d6433db34fea2d981"
server_key_iv: "c1db8e521f84e0e315c7ebcfa4068b3a"
server_key_version: "encryptionkey@0"
I20260928 02:09:42.716355 22793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:42.722325 22793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:42.726388 22793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:42.726643 22793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:42.746197 22793 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43727
I20260928 02:09:42.746248 22860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43727 every 8 connection(s)
I20260928 02:09:42.747671 22793 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:09:42.751400 22861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:42.756163 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 22793
I20260928 02:09:42.756333 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:09:42.756793 1754 external_mini_cluster.cc:1428] Setting key 9ace85ee3b117f174e19f11ed488f3ab
I20260928 02:09:42.762246 22861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a: Bootstrap starting.
I20260928 02:09:42.764648 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:42.758587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38531 (local address 127.0.0.1:43727)
0928 02:09:42.759461 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:42.759474 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:42.759513 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:09:42.761224 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:42.761245 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:42.761579 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:42.761953 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:42.763018 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:42.763024 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:42.763037 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:42.763052 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:42.763163 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:42.763197 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:42.763422 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:42.763561 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 02:09:42.763747 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":440,"threads_started":1}
I20260928 02:09:42.765702 22861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:42.766821 22861 log.cc:824] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:42.768456 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:09:42.769860 22861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a: No bootstrap required, opened a new log
I20260928 02:09:42.775166 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:42.772659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54225 (local address 127.0.0.1:43727)
0928 02:09:42.772840 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:42.772846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:42.773305 (+ 459us) server_negotiation.cc:407] Connection header received
0928 02:09:42.773437 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:42.773442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:42.773496 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:42.773606 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:42.774490 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:42.774496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:42.774502 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:42.774526 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:42.774588 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:42.774608 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:42.774729 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:42.774784 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 02:09:42.774838 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:09:42.778173 22861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba82340d6aa4717a05002cd4e8f248a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43727 } }
I20260928 02:09:42.778527 22861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:42.778628 22861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba82340d6aa4717a05002cd4e8f248a, State: Initialized, Role: FOLLOWER
I20260928 02:09:42.779285 22861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba82340d6aa4717a05002cd4e8f248a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43727 } }
I20260928 02:09:42.779488 22861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:09:42.779661 22861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:09:42.779844 22861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:42.782047 22861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba82340d6aa4717a05002cd4e8f248a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43727 } }
I20260928 02:09:42.782531 22861 leader_election.cc:302] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ba82340d6aa4717a05002cd4e8f248a; no voters:
I20260928 02:09:42.783174 22861 leader_election.cc:288] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:09:42.783303 22871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:42.783970 22871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a [term 1 LEADER]: Becoming Leader. State: Replica: 2ba82340d6aa4717a05002cd4e8f248a, State: Running, Role: LEADER
I20260928 02:09:42.784464 22871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba82340d6aa4717a05002cd4e8f248a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43727 } }
I20260928 02:09:42.785183 22861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:09:42.787905 22873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ba82340d6aa4717a05002cd4e8f248a. Latest consensus state: current_term: 1 leader_uuid: "2ba82340d6aa4717a05002cd4e8f248a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba82340d6aa4717a05002cd4e8f248a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43727 } } }
I20260928 02:09:42.788316 22873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:42.789733 22879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:09:42.789582 22872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ba82340d6aa4717a05002cd4e8f248a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba82340d6aa4717a05002cd4e8f248a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43727 } } }
I20260928 02:09:42.789948 22872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:42.794420 22879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:09:42.803491 22879 catalog_manager.cc:1347] Generated new cluster ID: ba5da79bd9b24f439effe7d698dc8f81
I20260928 02:09:42.803617 22879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:09:42.818493 22879 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:09:42.818984 22879 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:09:42.837919 22879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ba82340d6aa4717a05002cd4e8f248a: Generated new TSK 0
I20260928 02:09:42.839251 22879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:09:42.852684 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 22793
W20260928 02:09:42.854215 22856 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38531
W20260928 02:09:42.855031 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 22793: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:09:42.855239 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 22793
2026-09-28T02:09:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1985 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:09:42 dist-test-slave-23bp krb5kdc[22892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:09:42 dist-test-slave-23bp krb5kdc[22892](info): set up 2 sockets
Sep 28 02:09:42 dist-test-slave-23bp krb5kdc[22892](info): commencing operation
krb5kdc: starting...
W20260928 02:09:43.990593 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:09:44 dist-test-slave-23bp krb5kdc[22892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:09:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:09:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:09:44.279148 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36435
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:44.638058 22905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:44.638376 22905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:44.638479 22905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:44.645943 22905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:09:44.646064 22905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:44.646111 22905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:44.646173 22905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:09:44.646217 22905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:09:44.656276 22905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:44.657632 22905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:44.658939 22905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:44.671032 22910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:44.671008 22913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:44.671020 22911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:44.672081 22905 server_base.cc:1034] running on GCE node
I20260928 02:09:44.672868 22905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:44.674320 22905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:44.675562 22905 hybrid_clock.cc:648] HybridClock initialized: now 1790561384675517 us; error 39 us; skew 500 ppm
I20260928 02:09:44.675968 22905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:44 dist-test-slave-23bp krb5kdc[22892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:44.679191 22905 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:09:44.680807 22905 webserver.cc:460] Webserver started at http://127.0.0.1:33681/ using document root <none> and password file <none>
I20260928 02:09:44.681548 22905 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:44.681695 22905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:44.682008 22905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:44.685125 22905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "d360af04228e4e1d85e5d6f4bb4e4b12"
format_stamp: "Formatted at 2026-09-28 02:09:44 on dist-test-slave-23bp"
server_key: "2764b550a893e389267bdd17e930bef8"
server_key_iv: "bf2fb6c0efbccc3891872f3114c60512"
server_key_version: "encryptionkey@0"
I20260928 02:09:44.685920 22905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "d360af04228e4e1d85e5d6f4bb4e4b12"
format_stamp: "Formatted at 2026-09-28 02:09:44 on dist-test-slave-23bp"
server_key: "2764b550a893e389267bdd17e930bef8"
server_key_iv: "bf2fb6c0efbccc3891872f3114c60512"
server_key_version: "encryptionkey@0"
I20260928 02:09:44.691632 22905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:09:44.695425 22920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:44.696784 22905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:09:44.696974 22905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "d360af04228e4e1d85e5d6f4bb4e4b12"
format_stamp: "Formatted at 2026-09-28 02:09:44 on dist-test-slave-23bp"
server_key: "2764b550a893e389267bdd17e930bef8"
server_key_iv: "bf2fb6c0efbccc3891872f3114c60512"
server_key_version: "encryptionkey@0"
I20260928 02:09:44.697177 22905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:44.717052 22905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:44.721370 22905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:44.721748 22905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:44.742257 22905 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36435
I20260928 02:09:44.742358 22971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36435 every 8 connection(s)
I20260928 02:09:44.743731 22905 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:09:44.746505 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 22905
I20260928 02:09:44.746706 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:09:44.747268 1754 external_mini_cluster.cc:1428] Setting key 0d4e9f7a82b9c9a30c51f73dc31a94d2
I20260928 02:09:44.748091 22972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 02:09:44 dist-test-slave-23bp krb5kdc[22892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:09:44.762717 22972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12: Bootstrap starting.
I20260928 02:09:44.766482 22972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:44.767797 22972 log.cc:824] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:44.771355 22972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12: No bootstrap required, opened a new log
I20260928 02:09:44.772125 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:44.749095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:43131 (local address 127.0.0.1:36435)
0928 02:09:44.749939 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:44.749950 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:44.749991 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:09:44.751720 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:44.751743 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:44.752176 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:44.752514 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:44.754367 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:44.756235 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:44.758038 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:44.758810 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:44.758971 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:44.763997 (+ 5026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:44.764005 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:44.764019 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:44.764059 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:44.767235 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:44.768286 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:44.768294 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:44.768300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:44.768624 (+ 324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:44.769174 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:44.769180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:44.769183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:44.769795 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:44.770020 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:44.770595 (+ 575us) server_negotiation.cc:299] Negotiation successful
0928 02:09:44.770934 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":423,"threads_started":1}
I20260928 02:09:44.777034 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:09:44.781749 22972 raft_consensus.cc:348] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d360af04228e4e1d85e5d6f4bb4e4b12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36435 } }
I20260928 02:09:44.782202 22972 raft_consensus.cc:374] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:44.782330 22972 raft_consensus.cc:729] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d360af04228e4e1d85e5d6f4bb4e4b12, State: Initialized, Role: FOLLOWER
I20260928 02:09:44.783072 22972 consensus_queue.cc:260] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d360af04228e4e1d85e5d6f4bb4e4b12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36435 } }
I20260928 02:09:44.783367 22972 raft_consensus.cc:388] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:09:44.783542 22972 raft_consensus.cc:482] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:09:44.783735 22972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:44.786012 22972 raft_consensus.cc:504] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d360af04228e4e1d85e5d6f4bb4e4b12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36435 } }
I20260928 02:09:44.786514 22972 leader_election.cc:302] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d360af04228e4e1d85e5d6f4bb4e4b12; no voters:
I20260928 02:09:44.787185 22972 leader_election.cc:288] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:09:44.787298 22982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:44.789829 22972 sys_catalog.cc:564] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:09:44.791352 22982 raft_consensus.cc:686] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12 [term 1 LEADER]: Becoming Leader. State: Replica: d360af04228e4e1d85e5d6f4bb4e4b12, State: Running, Role: LEADER
I20260928 02:09:44.792521 22982 consensus_queue.cc:237] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d360af04228e4e1d85e5d6f4bb4e4b12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36435 } }
I20260928 02:09:44.795919 22984 sys_catalog.cc:455] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d360af04228e4e1d85e5d6f4bb4e4b12. Latest consensus state: current_term: 1 leader_uuid: "d360af04228e4e1d85e5d6f4bb4e4b12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d360af04228e4e1d85e5d6f4bb4e4b12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36435 } } }
I20260928 02:09:44.796399 22984 sys_catalog.cc:458] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:44.797433 22983 sys_catalog.cc:455] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d360af04228e4e1d85e5d6f4bb4e4b12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d360af04228e4e1d85e5d6f4bb4e4b12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36435 } } }
I20260928 02:09:44.797786 22983 sys_catalog.cc:458] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:44.800302 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:44.782399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:42363 (local address 127.0.0.1:36435)
0928 02:09:44.782570 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:44.782576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:44.783264 (+ 688us) server_negotiation.cc:407] Connection header received
0928 02:09:44.783474 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:44.783478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:44.783599 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:44.783717 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:44.785164 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:44.785929 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:44.787421 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:44.788137 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:44.788261 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:44.790946 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:44.790952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:44.790955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:44.790987 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:44.795221 (+ 4234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:44.796308 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:44.796313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:44.796317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:44.796613 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:44.797162 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:44.797168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:44.797174 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:44.797576 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:44.797983 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:44.799506 (+ 1523us) server_negotiation.cc:299] Negotiation successful
0928 02:09:44.799750 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:09:44.802419 22993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:09:44.807585 22993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:09:44.816956 22993 catalog_manager.cc:1347] Generated new cluster ID: 9ad5126e618240e89ba9454155d33a00
I20260928 02:09:44.817102 22993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:09:44.825183 22993 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:09:44.827059 22993 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:09:44.843622 22993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d360af04228e4e1d85e5d6f4bb4e4b12: Generated new TSK 0
I20260928 02:09:44.844554 22993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:09:44.859519 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 22905
I20260928 02:09:44.860795 22967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:44.954154 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 22905
W20260928 02:09:44.971318 22890 connection.cc:535] client connection to 127.0.0.1:36435 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36435): Connection reset by peer (error 104)
2026-09-28T02:09:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2098 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:09:45 dist-test-slave-23bp krb5kdc[23005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:09:45 dist-test-slave-23bp krb5kdc[23005](info): set up 2 sockets
Sep 28 02:09:45 dist-test-slave-23bp krb5kdc[23005](info): commencing operation
krb5kdc: starting...
W20260928 02:09:46.074004 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:09:46 dist-test-slave-23bp krb5kdc[23005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:09:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:09:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:09:46.336294 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41921
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:46.676239 23018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:46.676539 23018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:46.676712 23018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:46.684288 23018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:09:46.684460 23018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:46.684574 23018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:46.684680 23018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:09:46.684765 23018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:09:46.694866 23018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:46.696316 23018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:46.697683 23018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:46.709633 23026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:46.709730 23023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:46.709611 23024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:46.715939 23018 server_base.cc:1034] running on GCE node
I20260928 02:09:46.716794 23018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:46.718243 23018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:46.719472 23018 hybrid_clock.cc:648] HybridClock initialized: now 1790561386719431 us; error 58 us; skew 500 ppm
I20260928 02:09:46.719846 23018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:46 dist-test-slave-23bp krb5kdc[23005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:46.722949 23018 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:09:46.724507 23018 webserver.cc:460] Webserver started at http://127.0.0.1:44475/ using document root <none> and password file <none>
I20260928 02:09:46.725232 23018 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:46.725378 23018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:46.725695 23018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:46.728772 23018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "c279691fba2c45f4826d2aeaa76b6658"
format_stamp: "Formatted at 2026-09-28 02:09:46 on dist-test-slave-23bp"
server_key: "92758f74a153e8796967cb5e18cb64ed"
server_key_iv: "1fcf845289c0767d77a31e41ecca42c7"
server_key_version: "encryptionkey@0"
I20260928 02:09:46.729523 23018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "c279691fba2c45f4826d2aeaa76b6658"
format_stamp: "Formatted at 2026-09-28 02:09:46 on dist-test-slave-23bp"
server_key: "92758f74a153e8796967cb5e18cb64ed"
server_key_iv: "1fcf845289c0767d77a31e41ecca42c7"
server_key_version: "encryptionkey@0"
I20260928 02:09:46.734834 23018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:09:46.738700 23033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:46.740123 23018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:09:46.740311 23018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "c279691fba2c45f4826d2aeaa76b6658"
format_stamp: "Formatted at 2026-09-28 02:09:46 on dist-test-slave-23bp"
server_key: "92758f74a153e8796967cb5e18cb64ed"
server_key_iv: "1fcf845289c0767d77a31e41ecca42c7"
server_key_version: "encryptionkey@0"
I20260928 02:09:46.740518 23018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:46.760588 23018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:46.764736 23018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:46.765031 23018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:46.783644 23018 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41921
I20260928 02:09:46.783681 23084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41921 every 8 connection(s)
I20260928 02:09:46.785051 23018 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:09:46.788736 23085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:46.791738 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 23018
I20260928 02:09:46.791930 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:09:46.792335 1754 external_mini_cluster.cc:1428] Setting key b85fa55e8b79c253434de17432e14ec7
I20260928 02:09:46.800652 23085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658: Bootstrap starting.
I20260928 02:09:46.804355 23085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:46.805522 23085 log.cc:824] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:46.805995 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:46.794108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:59257 (local address 127.0.0.1:41921)
0928 02:09:46.794900 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:46.794912 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:46.794953 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:09:46.796676 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:46.796698 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:46.797127 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:46.797458 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:46.798105 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:46.799484 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:46.801397 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:46.802059 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:46.802214 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:46.803636 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:46.803643 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:46.803656 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:46.804189 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:46.804269 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:46.804294 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:46.804414 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:46.804785 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 02:09:46.804968 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":365,"threads_started":1}
I20260928 02:09:46.808588 23085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658: No bootstrap required, opened a new log
I20260928 02:09:46.810143 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:09:46.819242 23085 raft_consensus.cc:348] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c279691fba2c45f4826d2aeaa76b6658" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41921 } }
I20260928 02:09:46.819664 23085 raft_consensus.cc:374] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:46.819767 23085 raft_consensus.cc:729] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c279691fba2c45f4826d2aeaa76b6658, State: Initialized, Role: FOLLOWER
I20260928 02:09:46.820474 23085 consensus_queue.cc:260] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c279691fba2c45f4826d2aeaa76b6658" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41921 } }
I20260928 02:09:46.820739 23085 raft_consensus.cc:388] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:09:46.820906 23085 raft_consensus.cc:482] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:09:46.821102 23085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:46.822517 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:46.815632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:52855 (local address 127.0.0.1:41921)
0928 02:09:46.815829 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:46.815835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:46.816313 (+ 478us) server_negotiation.cc:407] Connection header received
0928 02:09:46.816468 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:46.816473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:46.816562 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:46.816684 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:46.817259 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:46.818435 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:46.819910 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:46.820395 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:46.820536 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:46.821741 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:46.821757 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:46.821760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:46.821778 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:46.821839 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:46.821855 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:46.821998 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:46.822185 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 02:09:46.822242 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:09:46.824126 23085 raft_consensus.cc:504] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c279691fba2c45f4826d2aeaa76b6658" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41921 } }
I20260928 02:09:46.824638 23085 leader_election.cc:302] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c279691fba2c45f4826d2aeaa76b6658; no voters:
I20260928 02:09:46.825193 23085 leader_election.cc:288] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:09:46.825314 23096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:46.826007 23096 raft_consensus.cc:686] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658 [term 1 LEADER]: Becoming Leader. State: Replica: c279691fba2c45f4826d2aeaa76b6658, State: Running, Role: LEADER
I20260928 02:09:46.826493 23096 consensus_queue.cc:237] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c279691fba2c45f4826d2aeaa76b6658" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41921 } }
I20260928 02:09:46.827414 23085 sys_catalog.cc:564] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:09:46.829216 23098 sys_catalog.cc:455] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c279691fba2c45f4826d2aeaa76b6658. Latest consensus state: current_term: 1 leader_uuid: "c279691fba2c45f4826d2aeaa76b6658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c279691fba2c45f4826d2aeaa76b6658" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41921 } } }
I20260928 02:09:46.829589 23098 sys_catalog.cc:458] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:46.830029 23097 sys_catalog.cc:455] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c279691fba2c45f4826d2aeaa76b6658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c279691fba2c45f4826d2aeaa76b6658" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41921 } } }
I20260928 02:09:46.830386 23097 sys_catalog.cc:458] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:46.832439 23101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:09:46.840503 23101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:09:46.849555 23101 catalog_manager.cc:1347] Generated new cluster ID: 214174be48cd49e4a15ab5ce488a50e0
I20260928 02:09:46.849679 23101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:09:46.873355 23101 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:09:46.875352 23101 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:09:46.889393 23101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c279691fba2c45f4826d2aeaa76b6658: Generated new TSK 0
I20260928 02:09:46.890478 23101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:09:46.908572 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 23018
W20260928 02:09:46.909773 23080 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.150:59257
W20260928 02:09:46.910547 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 23018: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:09:46.910702 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 23018
W20260928 02:09:46.927731 23003 connection.cc:535] client connection to 127.0.0.1:41921 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41921): Connection reset by peer (error 104)
2026-09-28T02:09:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1956 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:09:47 dist-test-slave-23bp krb5kdc[23117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:09:47 dist-test-slave-23bp krb5kdc[23117](info): set up 2 sockets
Sep 28 02:09:47 dist-test-slave-23bp krb5kdc[23117](info): commencing operation
krb5kdc: starting...
W20260928 02:09:48.029986 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:09:48 dist-test-slave-23bp krb5kdc[23117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:09:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:09:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:09:48.311012 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38629
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:48.662218 23130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:48.662564 23130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:48.662734 23130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:48.670660 23130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:09:48.670792 23130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:48.670878 23130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:48.670969 23130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:09:48.671106 23130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:09:48.681809 23130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:48.683306 23130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:48.684823 23130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:48.698259 23138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:48.698333 23136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:48.698372 23135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:48.699465 23130 server_base.cc:1034] running on GCE node
I20260928 02:09:48.700551 23130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:48.702310 23130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:48.703600 23130 hybrid_clock.cc:648] HybridClock initialized: now 1790561388703540 us; error 59 us; skew 500 ppm
I20260928 02:09:48.704078 23130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:48 dist-test-slave-23bp krb5kdc[23117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:48.714473 23130 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:09:48.716404 23130 webserver.cc:460] Webserver started at http://127.0.0.1:33969/ using document root <none> and password file <none>
I20260928 02:09:48.717216 23130 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:48.717383 23130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:48.717711 23130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:48.721140 23130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "49d77ce639c74bf6973122b75178fafa"
format_stamp: "Formatted at 2026-09-28 02:09:48 on dist-test-slave-23bp"
server_key: "2d8be4cf9273d1e6844eb24a19028371"
server_key_iv: "a6cf3de421e3d26f56f89fe894a6a9db"
server_key_version: "encryptionkey@0"
I20260928 02:09:48.722007 23130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "49d77ce639c74bf6973122b75178fafa"
format_stamp: "Formatted at 2026-09-28 02:09:48 on dist-test-slave-23bp"
server_key: "2d8be4cf9273d1e6844eb24a19028371"
server_key_iv: "a6cf3de421e3d26f56f89fe894a6a9db"
server_key_version: "encryptionkey@0"
I20260928 02:09:48.728039 23130 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 02:09:48.732260 23145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:48.733839 23130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:09:48.734068 23130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "49d77ce639c74bf6973122b75178fafa"
format_stamp: "Formatted at 2026-09-28 02:09:48 on dist-test-slave-23bp"
server_key: "2d8be4cf9273d1e6844eb24a19028371"
server_key_iv: "a6cf3de421e3d26f56f89fe894a6a9db"
server_key_version: "encryptionkey@0"
I20260928 02:09:48.734299 23130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:48.740767 23130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:48.745292 23130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:48.745661 23130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:48.764770 23130 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38629
I20260928 02:09:48.764806 23196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38629 every 8 connection(s)
I20260928 02:09:48.766299 23130 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:09:48.769655 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 23130
I20260928 02:09:48.769838 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:09:48.770393 1754 external_mini_cluster.cc:1428] Setting key 07a1cee5b859fbccae6498603328a95b
I20260928 02:09:48.770591 23197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:48.778928 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:48.772269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:44121 (local address 127.0.0.1:38629)
0928 02:09:48.773090 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:48.773102 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:48.773144 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:09:48.775419 (+ 2275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:48.775448 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:48.775815 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:48.776254 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:48.777322 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:48.777328 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:48.777341 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:48.777363 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:48.777421 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:48.777449 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:48.777702 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:48.777845 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 02:09:48.778026 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":335,"threads_started":1}
I20260928 02:09:48.782610 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:09:48.783957 23197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa: Bootstrap starting.
I20260928 02:09:48.787791 23197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:48.789022 23197 log.cc:824] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:48.789994 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:48.787476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:59851 (local address 127.0.0.1:38629)
0928 02:09:48.787706 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:48.787712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:48.787859 (+ 147us) server_negotiation.cc:407] Connection header received
0928 02:09:48.788058 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:48.788063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:48.788127 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:48.788239 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:48.789322 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:48.789326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:48.789329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:48.789345 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:48.789385 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:48.789426 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:48.789557 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:48.789651 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 02:09:48.789724 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:09:48.792179 23197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa: No bootstrap required, opened a new log
I20260928 02:09:48.800185 23197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d77ce639c74bf6973122b75178fafa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38629 } }
I20260928 02:09:48.800616 23197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:48.800717 23197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d77ce639c74bf6973122b75178fafa, State: Initialized, Role: FOLLOWER
I20260928 02:09:48.801328 23197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d77ce639c74bf6973122b75178fafa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38629 } }
I20260928 02:09:48.801573 23197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:09:48.801724 23197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:09:48.801868 23197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:48.804256 23197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d77ce639c74bf6973122b75178fafa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38629 } }
I20260928 02:09:48.804774 23197 leader_election.cc:302] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49d77ce639c74bf6973122b75178fafa; no voters:
I20260928 02:09:48.805356 23197 leader_election.cc:288] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:09:48.805521 23207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:48.806183 23207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa [term 1 LEADER]: Becoming Leader. State: Replica: 49d77ce639c74bf6973122b75178fafa, State: Running, Role: LEADER
I20260928 02:09:48.806679 23207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d77ce639c74bf6973122b75178fafa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38629 } }
I20260928 02:09:48.807519 23197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:09:48.809710 23209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49d77ce639c74bf6973122b75178fafa. Latest consensus state: current_term: 1 leader_uuid: "49d77ce639c74bf6973122b75178fafa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d77ce639c74bf6973122b75178fafa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38629 } } }
I20260928 02:09:48.810189 23209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:48.811092 23208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49d77ce639c74bf6973122b75178fafa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d77ce639c74bf6973122b75178fafa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38629 } } }
I20260928 02:09:48.811448 23208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:48.812840 23214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:09:48.818305 23214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:09:48.827821 23214 catalog_manager.cc:1347] Generated new cluster ID: c44720184db84849acdf51d8b009910b
I20260928 02:09:48.827979 23214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:09:48.844286 23214 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:09:48.844812 23214 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:09:48.860353 23214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49d77ce639c74bf6973122b75178fafa: Generated new TSK 0
I20260928 02:09:48.861536 23214 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:09:48.873674 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 23130
W20260928 02:09:48.875178 23192 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.150:44121
W20260928 02:09:48.875970 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 23130: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:09:48.876130 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 23130
2026-09-28T02:09:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1965 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:09:48 dist-test-slave-23bp krb5kdc[23228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:09:48 dist-test-slave-23bp krb5kdc[23228](info): set up 2 sockets
Sep 28 02:09:48 dist-test-slave-23bp krb5kdc[23228](info): commencing operation
krb5kdc: starting...
W20260928 02:09:50.022399 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:09:50 dist-test-slave-23bp krb5kdc[23228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:09:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:09:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:09:50.284227 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42259
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:50.619820 23241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:50.620187 23241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:50.620355 23241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:50.627889 23241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:09:50.628050 23241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:50.628162 23241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:50.628271 23241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:09:50.628350 23241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:09:50.638908 23241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42259
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:50.640530 23241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:50.642015 23241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:50.654448 23248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:50.654551 23250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:50.654510 23247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:50.655525 23241 server_base.cc:1034] running on GCE node
I20260928 02:09:50.656291 23241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:50.657655 23241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:50.658887 23241 hybrid_clock.cc:648] HybridClock initialized: now 1790561390658836 us; error 48 us; skew 500 ppm
I20260928 02:09:50.659312 23241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:50 dist-test-slave-23bp krb5kdc[23228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:50.662364 23241 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:09:50.663857 23241 webserver.cc:460] Webserver started at http://127.0.0.1:39273/ using document root <none> and password file <none>
I20260928 02:09:50.664541 23241 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:50.664636 23241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:50.664886 23241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:50.667951 23241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "b7b1e78a33e049a284ca28ca96521105"
format_stamp: "Formatted at 2026-09-28 02:09:50 on dist-test-slave-23bp"
server_key: "8587d6030ccf54ee864512d13b777f09"
server_key_iv: "23481ba00d72fc59a4d73560b9496c48"
server_key_version: "encryptionkey@0"
I20260928 02:09:50.668759 23241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "b7b1e78a33e049a284ca28ca96521105"
format_stamp: "Formatted at 2026-09-28 02:09:50 on dist-test-slave-23bp"
server_key: "8587d6030ccf54ee864512d13b777f09"
server_key_iv: "23481ba00d72fc59a4d73560b9496c48"
server_key_version: "encryptionkey@0"
I20260928 02:09:50.674043 23241 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:09:50.677539 23257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:50.679248 23241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:09:50.679427 23241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "b7b1e78a33e049a284ca28ca96521105"
format_stamp: "Formatted at 2026-09-28 02:09:50 on dist-test-slave-23bp"
server_key: "8587d6030ccf54ee864512d13b777f09"
server_key_iv: "23481ba00d72fc59a4d73560b9496c48"
server_key_version: "encryptionkey@0"
I20260928 02:09:50.679637 23241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:50.703043 23241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:50.707358 23241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:50.707669 23241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:50.726251 23241 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42259
I20260928 02:09:50.726284 23309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42259 every 8 connection(s)
I20260928 02:09:50.727708 23241 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:09:50.729528 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 23241
I20260928 02:09:50.729704 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:09:50.730144 1754 external_mini_cluster.cc:1428] Setting key afadfc2926e57ec4ac6f38fb115d5523
I20260928 02:09:50.732182 23310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:50.743968 23310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105: Bootstrap starting.
Sep 28 02:09:50 dist-test-slave-23bp krb5kdc[23228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:09:50.747928 23310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:50.749037 23310 log.cc:824] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:50.751952 23310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105: No bootstrap required, opened a new log
I20260928 02:09:50.753408 23313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:50.731902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:39303 (local address 127.0.0.1:42259)
0928 02:09:50.732701 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:50.732719 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:50.733074 (+ 355us) server_negotiation.cc:407] Connection header received
0928 02:09:50.734892 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:50.734913 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:50.735386 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:50.735792 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:50.737233 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:50.738568 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:50.739932 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:50.740788 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:50.741020 (+ 232us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:50.745790 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:50.745799 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:50.745816 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:50.745851 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:50.749078 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:50.749993 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:50.750000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:50.750006 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:50.750178 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:50.750672 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:50.750675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:50.750677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:50.751271 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:50.751600 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:50.752143 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 02:09:50.752445 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":359,"threads_started":1}
I20260928 02:09:50.757542 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:09:50.760793 23310 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b1e78a33e049a284ca28ca96521105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42259 } }
I20260928 02:09:50.761168 23310 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:50.761278 23310 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b1e78a33e049a284ca28ca96521105, State: Initialized, Role: FOLLOWER
I20260928 02:09:50.761894 23310 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b1e78a33e049a284ca28ca96521105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42259 } }
I20260928 02:09:50.762151 23310 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:09:50.762305 23310 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:09:50.762483 23310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:50.765236 23310 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b1e78a33e049a284ca28ca96521105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42259 } }
I20260928 02:09:50.765888 23310 leader_election.cc:302] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7b1e78a33e049a284ca28ca96521105; no voters:
I20260928 02:09:50.766521 23310 leader_election.cc:288] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:09:50.766660 23320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:50.767366 23320 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105 [term 1 LEADER]: Becoming Leader. State: Replica: b7b1e78a33e049a284ca28ca96521105, State: Running, Role: LEADER
I20260928 02:09:50.768118 23320 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b1e78a33e049a284ca28ca96521105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42259 } }
I20260928 02:09:50.769385 23310 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:09:50.771246 23321 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7b1e78a33e049a284ca28ca96521105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b1e78a33e049a284ca28ca96521105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42259 } } }
I20260928 02:09:50.771154 23322 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7b1e78a33e049a284ca28ca96521105. Latest consensus state: current_term: 1 leader_uuid: "b7b1e78a33e049a284ca28ca96521105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b1e78a33e049a284ca28ca96521105" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42259 } } }
I20260928 02:09:50.771610 23322 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:50.774475 23321 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:50.778981 23325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:09:50.781466 23313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:50.761880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:49615 (local address 127.0.0.1:42259)
0928 02:09:50.762085 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:50.762093 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:50.762661 (+ 568us) server_negotiation.cc:407] Connection header received
0928 02:09:50.762840 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:50.762848 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:50.762931 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:50.763036 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:50.764481 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:50.765630 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:50.767399 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:50.768139 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:50.768269 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:50.771975 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:50.771985 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:50.771993 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:50.772058 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:50.777215 (+ 5157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:50.779122 (+ 1907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:50.779127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:50.779131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:50.779340 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:50.779866 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:50.779871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:50.779875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:50.780172 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:50.780489 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:50.780868 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 02:09:50.781053 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:09:50.784691 23325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:09:50.793848 23325 catalog_manager.cc:1347] Generated new cluster ID: 1541ef8a828b4796adecec6f95ecb761
I20260928 02:09:50.793967 23325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:09:50.815599 23325 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:09:50.816962 23325 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:09:50.830828 23325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7b1e78a33e049a284ca28ca96521105: Generated new TSK 0
I20260928 02:09:50.832016 23325 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:09:50.843189 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 23241
I20260928 02:09:50.844448 23305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:50.936957 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 23241
W20260928 02:09:50.954329 23226 connection.cc:535] client connection to 127.0.0.1:42259 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42259): Connection reset by peer (error 104)
2026-09-28T02:09:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2061 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:09:51 dist-test-slave-23bp krb5kdc[23342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:09:51 dist-test-slave-23bp krb5kdc[23342](info): set up 2 sockets
Sep 28 02:09:51 dist-test-slave-23bp krb5kdc[23342](info): commencing operation
krb5kdc: starting...
W20260928 02:09:52.058827 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:09:52 dist-test-slave-23bp krb5kdc[23342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:09:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:09:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:09:52.327158 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43293
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:52.671620 23355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:52.671973 23355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:52.672137 23355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:52.680094 23355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:09:52.680239 23355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:52.680347 23355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:52.680437 23355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:09:52.680503 23355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:09:52.690385 23355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43293
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:52.691787 23355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:52.693099 23355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:52.705084 23360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:52.705128 23361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:52.705152 23363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:52.708575 23355 server_base.cc:1034] running on GCE node
I20260928 02:09:52.709304 23355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:52.710704 23355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:52.711931 23355 hybrid_clock.cc:648] HybridClock initialized: now 1790561392711889 us; error 35 us; skew 500 ppm
I20260928 02:09:52.712283 23355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:52 dist-test-slave-23bp krb5kdc[23342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:52.715438 23355 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:09:52.716827 23355 webserver.cc:460] Webserver started at http://127.0.0.1:37089/ using document root <none> and password file <none>
I20260928 02:09:52.717518 23355 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:52.717661 23355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:52.717967 23355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:52.720925 23355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "e968a3f241f64feea336feb2596a89f9"
format_stamp: "Formatted at 2026-09-28 02:09:52 on dist-test-slave-23bp"
server_key: "0d3488a800dad33a776f2931031507de"
server_key_iv: "88f8cd46091cfe14a8543b6336afb405"
server_key_version: "encryptionkey@0"
I20260928 02:09:52.721745 23355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "e968a3f241f64feea336feb2596a89f9"
format_stamp: "Formatted at 2026-09-28 02:09:52 on dist-test-slave-23bp"
server_key: "0d3488a800dad33a776f2931031507de"
server_key_iv: "88f8cd46091cfe14a8543b6336afb405"
server_key_version: "encryptionkey@0"
I20260928 02:09:52.726732 23355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:09:52.730057 23370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:52.731482 23355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:09:52.731647 23355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "e968a3f241f64feea336feb2596a89f9"
format_stamp: "Formatted at 2026-09-28 02:09:52 on dist-test-slave-23bp"
server_key: "0d3488a800dad33a776f2931031507de"
server_key_iv: "88f8cd46091cfe14a8543b6336afb405"
server_key_version: "encryptionkey@0"
I20260928 02:09:52.731837 23355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:52.754804 23355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:52.759305 23355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:52.759613 23355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:52.778439 23355 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43293
I20260928 02:09:52.778460 23421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43293 every 8 connection(s)
I20260928 02:09:52.779830 23355 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:09:52.782374 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 23355
I20260928 02:09:52.782565 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:09:52.783022 1754 external_mini_cluster.cc:1428] Setting key 271ea2822af0f9105d45031b293f2df4
I20260928 02:09:52.783756 23422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:52.797582 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:52.784911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:47123 (local address 127.0.0.1:43293)
0928 02:09:52.785672 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:52.785690 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:52.785740 (+ 50us) server_negotiation.cc:407] Connection header received
0928 02:09:52.787515 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:52.787542 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:52.788067 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:52.788500 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:52.789006 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:52.791017 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:52.792280 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:52.793287 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:52.793456 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:52.794754 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:52.794766 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:52.794789 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:52.795584 (+ 795us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:52.795677 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:52.795706 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:52.795880 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:52.796130 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 02:09:52.796312 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":303,"threads_started":1}
I20260928 02:09:52.798735 23422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9: Bootstrap starting.
I20260928 02:09:52.801368 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:09:52.802148 23422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:52.803185 23422 log.cc:824] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:52.806303 23422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9: No bootstrap required, opened a new log
I20260928 02:09:52.812311 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:52.805857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:33155 (local address 127.0.0.1:43293)
0928 02:09:52.806076 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:52.806082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:52.806362 (+ 280us) server_negotiation.cc:407] Connection header received
0928 02:09:52.806565 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:52.806569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:52.806627 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:52.806736 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:52.807298 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:52.808134 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:52.809450 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:52.810111 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:52.810264 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:52.811529 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:52.811534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:52.811537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:52.811556 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:52.811594 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:52.811609 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:52.811694 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:52.811912 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 02:09:52.811975 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:09:52.814949 23422 raft_consensus.cc:348] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e968a3f241f64feea336feb2596a89f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43293 } }
I20260928 02:09:52.815397 23422 raft_consensus.cc:374] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:52.815518 23422 raft_consensus.cc:729] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e968a3f241f64feea336feb2596a89f9, State: Initialized, Role: FOLLOWER
I20260928 02:09:52.816082 23422 consensus_queue.cc:260] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e968a3f241f64feea336feb2596a89f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43293 } }
I20260928 02:09:52.816335 23422 raft_consensus.cc:388] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:09:52.816485 23422 raft_consensus.cc:482] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:09:52.816668 23422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:52.818897 23422 raft_consensus.cc:504] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e968a3f241f64feea336feb2596a89f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43293 } }
I20260928 02:09:52.819429 23422 leader_election.cc:302] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e968a3f241f64feea336feb2596a89f9; no voters:
I20260928 02:09:52.819995 23422 leader_election.cc:288] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:09:52.820137 23432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:52.820829 23432 raft_consensus.cc:686] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9 [term 1 LEADER]: Becoming Leader. State: Replica: e968a3f241f64feea336feb2596a89f9, State: Running, Role: LEADER
I20260928 02:09:52.821370 23432 consensus_queue.cc:237] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e968a3f241f64feea336feb2596a89f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43293 } }
I20260928 02:09:52.822065 23422 sys_catalog.cc:564] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:09:52.824167 23433 sys_catalog.cc:455] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e968a3f241f64feea336feb2596a89f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e968a3f241f64feea336feb2596a89f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43293 } } }
I20260928 02:09:52.824576 23433 sys_catalog.cc:458] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:52.827196 23440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:09:52.827005 23434 sys_catalog.cc:455] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e968a3f241f64feea336feb2596a89f9. Latest consensus state: current_term: 1 leader_uuid: "e968a3f241f64feea336feb2596a89f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e968a3f241f64feea336feb2596a89f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43293 } } }
I20260928 02:09:52.827419 23434 sys_catalog.cc:458] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:52.832353 23440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:09:52.841732 23440 catalog_manager.cc:1347] Generated new cluster ID: 886383f9ffda4f16b2c6e702ae7e96cb
I20260928 02:09:52.841849 23440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:09:52.852310 23440 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:09:52.854096 23440 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:09:52.874470 23440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e968a3f241f64feea336feb2596a89f9: Generated new TSK 0
I20260928 02:09:52.875454 23440 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:09:52.890975 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 23355
W20260928 02:09:52.892282 23417 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.150:47123
W20260928 02:09:52.892990 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 23355: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:09:52.893142 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 23355
W20260928 02:09:52.914425 23340 connection.cc:535] client connection to 127.0.0.1:43293 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43293): Connection reset by peer (error 104)
2026-09-28T02:09:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1959 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:09:52 dist-test-slave-23bp krb5kdc[23453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:09:52 dist-test-slave-23bp krb5kdc[23453](info): set up 2 sockets
Sep 28 02:09:52 dist-test-slave-23bp krb5kdc[23453](info): commencing operation
krb5kdc: starting...
W20260928 02:09:54.025990 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:09:54 dist-test-slave-23bp krb5kdc[23453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:09:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:09:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:09:54.298300 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37769
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:54.623572 23466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:54.623951 23466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:54.624130 23466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:54.631366 23466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:09:54.631538 23466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:54.631656 23466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:54.631771 23466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:09:54.631861 23466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:09:54.641952 23466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37769
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:54.643399 23466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:54.644776 23466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:54.657079 23471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:54.656998 23472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:54.657207 23474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:54.658293 23466 server_base.cc:1034] running on GCE node
I20260928 02:09:54.659216 23466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:54.660646 23466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:54.661938 23466 hybrid_clock.cc:648] HybridClock initialized: now 1790561394661853 us; error 52 us; skew 500 ppm
I20260928 02:09:54.662307 23466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:54 dist-test-slave-23bp krb5kdc[23453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:54.665448 23466 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:09:54.666996 23466 webserver.cc:460] Webserver started at http://127.0.0.1:33055/ using document root <none> and password file <none>
I20260928 02:09:54.667734 23466 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:54.667893 23466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:54.668207 23466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:54.671206 23466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "d9beaf4a3a17438c9fc4cb6149f449e4"
format_stamp: "Formatted at 2026-09-28 02:09:54 on dist-test-slave-23bp"
server_key: "3bd70942400e1a9b1a83c90429a48244"
server_key_iv: "113b4fb6ec960929c014fbff8013a687"
server_key_version: "encryptionkey@0"
I20260928 02:09:54.672053 23466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "d9beaf4a3a17438c9fc4cb6149f449e4"
format_stamp: "Formatted at 2026-09-28 02:09:54 on dist-test-slave-23bp"
server_key: "3bd70942400e1a9b1a83c90429a48244"
server_key_iv: "113b4fb6ec960929c014fbff8013a687"
server_key_version: "encryptionkey@0"
I20260928 02:09:54.677316 23466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:09:54.680802 23481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:54.682147 23466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:09:54.682332 23466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "d9beaf4a3a17438c9fc4cb6149f449e4"
format_stamp: "Formatted at 2026-09-28 02:09:54 on dist-test-slave-23bp"
server_key: "3bd70942400e1a9b1a83c90429a48244"
server_key_iv: "113b4fb6ec960929c014fbff8013a687"
server_key_version: "encryptionkey@0"
I20260928 02:09:54.682533 23466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:54.688400 23466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:54.692518 23466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:54.692830 23466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:54.711755 23466 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37769
I20260928 02:09:54.711796 23532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37769 every 8 connection(s)
I20260928 02:09:54.713281 23466 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:09:54.717079 23533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:54.722837 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 23466
I20260928 02:09:54.723027 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:09:54.723599 1754 external_mini_cluster.cc:1428] Setting key 11fd23686a2430b130a9e32e038ea86e
I20260928 02:09:54.728427 23533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4: Bootstrap starting.
I20260928 02:09:54.731515 23536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:54.725447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:47435 (local address 127.0.0.1:37769)
0928 02:09:54.726269 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:54.726281 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:54.726324 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:09:54.728060 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:54.728086 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:54.728401 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:54.728833 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:54.729891 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:54.729897 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:54.729913 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:54.729927 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:54.729985 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:54.730011 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:54.730248 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:54.730382 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 02:09:54.730566 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":351,"threads_started":1}
I20260928 02:09:54.731876 23533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:54.732923 23533 log.cc:824] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:54.735358 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:09:54.736147 23533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4: No bootstrap required, opened a new log
I20260928 02:09:54.742276 23536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:54.739747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:52527 (local address 127.0.0.1:37769)
0928 02:09:54.739991 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:54.739997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:54.740339 (+ 342us) server_negotiation.cc:407] Connection header received
0928 02:09:54.740517 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:54.740521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:54.740600 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:54.740681 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:54.741600 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:54.741604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:54.741607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:54.741622 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:54.741663 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:54.741676 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:54.741779 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:54.741898 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 02:09:54.741951 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:09:54.744855 23533 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9beaf4a3a17438c9fc4cb6149f449e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37769 } }
I20260928 02:09:54.745244 23533 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:54.745366 23533 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9beaf4a3a17438c9fc4cb6149f449e4, State: Initialized, Role: FOLLOWER
I20260928 02:09:54.746011 23533 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9beaf4a3a17438c9fc4cb6149f449e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37769 } }
I20260928 02:09:54.746222 23533 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:09:54.746323 23533 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:09:54.746449 23533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:54.748648 23533 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9beaf4a3a17438c9fc4cb6149f449e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37769 } }
I20260928 02:09:54.749075 23533 leader_election.cc:302] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9beaf4a3a17438c9fc4cb6149f449e4; no voters:
I20260928 02:09:54.749603 23533 leader_election.cc:288] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:09:54.749715 23543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:54.751138 23543 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4 [term 1 LEADER]: Becoming Leader. State: Replica: d9beaf4a3a17438c9fc4cb6149f449e4, State: Running, Role: LEADER
I20260928 02:09:54.751633 23543 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9beaf4a3a17438c9fc4cb6149f449e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37769 } }
I20260928 02:09:54.751881 23533 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:09:54.754789 23544 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9beaf4a3a17438c9fc4cb6149f449e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9beaf4a3a17438c9fc4cb6149f449e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37769 } } }
I20260928 02:09:54.755205 23544 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:54.755681 23545 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9beaf4a3a17438c9fc4cb6149f449e4. Latest consensus state: current_term: 1 leader_uuid: "d9beaf4a3a17438c9fc4cb6149f449e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9beaf4a3a17438c9fc4cb6149f449e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37769 } } }
I20260928 02:09:54.755962 23545 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4 [sys.catalog]: This master's current role is: LEADER
W20260928 02:09:54.766453 23559 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:09:54.766610 23559 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:09:54.766762 23551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:09:54.768446 23551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:09:54.776645 23551 catalog_manager.cc:1347] Generated new cluster ID: a4a458e320c2441494278625f3866c94
I20260928 02:09:54.776773 23551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:09:54.801890 23551 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:09:54.802345 23551 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:09:54.810050 23551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9beaf4a3a17438c9fc4cb6149f449e4: Generated new TSK 0
I20260928 02:09:54.810894 23551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:09:54.823189 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 23466
W20260928 02:09:54.824579 23528 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.150:47435
W20260928 02:09:54.825337 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 23466: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:09:54.825487 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 23466
2026-09-28T02:09:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1928 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:09:54 dist-test-slave-23bp krb5kdc[23564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:09:54 dist-test-slave-23bp krb5kdc[23564](info): set up 2 sockets
Sep 28 02:09:54 dist-test-slave-23bp krb5kdc[23564](info): commencing operation
krb5kdc: starting...
W20260928 02:09:55.945894 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:09:56 dist-test-slave-23bp krb5kdc[23564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:09:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:09:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:09:56.200270 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:46661
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:46661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:56.532660 23577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:56.533052 23577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:56.533221 23577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:56.540474 23577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:09:56.540578 23577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:56.540625 23577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:56.540686 23577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:09:56.540731 23577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:09:56.551210 23577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:46661
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:46661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:56.552526 23577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:56.553877 23577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:56.565737 23582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:56.565797 23583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:56.565821 23585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:56.566666 23577 server_base.cc:1034] running on GCE node
I20260928 02:09:56.567464 23577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:56.568801 23577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:56.570190 23577 hybrid_clock.cc:648] HybridClock initialized: now 1790561396570144 us; error 31 us; skew 500 ppm
I20260928 02:09:56.570521 23577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:56 dist-test-slave-23bp krb5kdc[23564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561396, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:56.573500 23577 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:09:56.575057 23577 webserver.cc:460] Webserver started at http://127.1.182.190:36589/ using document root <none> and password file <none>
I20260928 02:09:56.575732 23577 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:56.575831 23577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:56.576068 23577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:56.578902 23577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "30870b3190a840a78bbbab6f5920e757"
format_stamp: "Formatted at 2026-09-28 02:09:56 on dist-test-slave-23bp"
server_key: "5c284f27ce9ad46be286eaa3d0e6f3e0"
server_key_iv: "01cc5bab2555f0ee29627d6ba1cb8e9d"
server_key_version: "encryptionkey@0"
I20260928 02:09:56.579799 23577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "30870b3190a840a78bbbab6f5920e757"
format_stamp: "Formatted at 2026-09-28 02:09:56 on dist-test-slave-23bp"
server_key: "5c284f27ce9ad46be286eaa3d0e6f3e0"
server_key_iv: "01cc5bab2555f0ee29627d6ba1cb8e9d"
server_key_version: "encryptionkey@0"
I20260928 02:09:56.585018 23577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260928 02:09:56.588588 23592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:56.590488 23577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:09:56.590682 23577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "30870b3190a840a78bbbab6f5920e757"
format_stamp: "Formatted at 2026-09-28 02:09:56 on dist-test-slave-23bp"
server_key: "5c284f27ce9ad46be286eaa3d0e6f3e0"
server_key_iv: "01cc5bab2555f0ee29627d6ba1cb8e9d"
server_key_version: "encryptionkey@0"
I20260928 02:09:56.590974 23577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:56.615765 23577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:56.620170 23577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:56.620483 23577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:56.640089 23577 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:46661
I20260928 02:09:56.640096 23643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:46661 every 8 connection(s)
I20260928 02:09:56.641600 23577 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:09:56.645195 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 23577
I20260928 02:09:56.645404 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:09:56.645521 23644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:56.645959 1754 external_mini_cluster.cc:1428] Setting key 7602650de4b0fe41c8acc089faccd9ca
I20260928 02:09:56.660049 23644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757: Bootstrap starting.
Sep 28 02:09:56 dist-test-slave-23bp krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:09:56.663702 23644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:56.665132 23644 log.cc:824] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:56.668599 23644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757: No bootstrap required, opened a new log
I20260928 02:09:56.670701 23647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:56.647967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51391 (local address 127.1.182.190:46661)
0928 02:09:56.648848 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:56.648863 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:56.648911 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:09:56.650685 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:56.650715 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:56.651311 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:56.651759 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:56.653112 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:56.655106 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:56.656325 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:56.657442 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:56.657642 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:56.661793 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:56.661805 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:56.661824 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:56.661877 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:56.665963 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:56.666901 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:56.666909 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:56.666917 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:56.667141 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:56.667584 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:56.667588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:56.667591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:56.668254 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:56.668556 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:56.668917 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 02:09:56.669275 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":363,"threads_started":1}
I20260928 02:09:56.674772 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:09:56.677929 23644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30870b3190a840a78bbbab6f5920e757" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 46661 } }
I20260928 02:09:56.678306 23644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:56.678406 23644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30870b3190a840a78bbbab6f5920e757, State: Initialized, Role: FOLLOWER
I20260928 02:09:56.679123 23644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30870b3190a840a78bbbab6f5920e757" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 46661 } }
I20260928 02:09:56.679448 23644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:09:56.679600 23644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:09:56.679780 23644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:56.682001 23644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30870b3190a840a78bbbab6f5920e757" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 46661 } }
I20260928 02:09:56.682518 23644 leader_election.cc:302] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30870b3190a840a78bbbab6f5920e757; no voters:
I20260928 02:09:56.683207 23644 leader_election.cc:288] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:09:56.683293 23654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:56.685822 23644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:09:56.687354 23654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757 [term 1 LEADER]: Becoming Leader. State: Replica: 30870b3190a840a78bbbab6f5920e757, State: Running, Role: LEADER
I20260928 02:09:56.688107 23654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30870b3190a840a78bbbab6f5920e757" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 46661 } }
I20260928 02:09:56.692123 23655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30870b3190a840a78bbbab6f5920e757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30870b3190a840a78bbbab6f5920e757" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 46661 } } }
I20260928 02:09:56.692579 23655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:56.692600 23656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30870b3190a840a78bbbab6f5920e757. Latest consensus state: current_term: 1 leader_uuid: "30870b3190a840a78bbbab6f5920e757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30870b3190a840a78bbbab6f5920e757" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 46661 } } }
I20260928 02:09:56.693027 23656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757 [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:56.695407 23647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:56.679214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58063 (local address 127.1.182.190:46661)
0928 02:09:56.679391 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:56.679399 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:56.679912 (+ 513us) server_negotiation.cc:407] Connection header received
0928 02:09:56.680127 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:56.680133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:56.680206 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:56.680310 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:56.681764 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:56.682610 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:56.684216 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:56.684641 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:56.684786 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:56.687453 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:56.687460 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:56.687465 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:09:56.687502 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:56.690874 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:56.692080 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:56.692087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:56.692092 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:56.692271 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:09:56.693023 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:09:56.693031 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:09:56.693037 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:09:56.693611 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:56.693985 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:56.694581 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 02:09:56.694817 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":47}
I20260928 02:09:56.699404 23660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:09:56.705159 23660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:09:56.715438 23660 catalog_manager.cc:1347] Generated new cluster ID: c9a01e62119c433e8b250fc960272278
I20260928 02:09:56.715569 23660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:09:56.733686 23660 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:09:56.735023 23660 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:09:56.743650 23660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30870b3190a840a78bbbab6f5920e757: Generated new TSK 0
I20260928 02:09:56.744433 23660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:09:56.755273 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 23577
I20260928 02:09:56.756510 23639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:09:56.847649 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 23577
W20260928 02:09:56.864717 23562 connection.cc:535] client connection to 127.1.182.190:46661 recv error: Network error: failed to read from TLS socket (remote: 127.1.182.190:46661): Connection reset by peer (error 104)
2026-09-28T02:09:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2021 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:09:56 dist-test-slave-23bp krb5kdc[23676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:09:56 dist-test-slave-23bp krb5kdc[23676](info): set up 2 sockets
Sep 28 02:09:56 dist-test-slave-23bp krb5kdc[23676](info): commencing operation
krb5kdc: starting...
W20260928 02:09:58.041915 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:09:58 dist-test-slave-23bp krb5kdc[23676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:09:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:09:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:09:58.302151 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:38625
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:46311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:38625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:09:58.633303 23689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:09:58.633702 23689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:09:58.633867 23689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:09:58.641100 23689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:09:58.641234 23689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:09:58.641319 23689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:09:58.641403 23689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:09:58.641465 23689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:09:58.651333 23689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:46311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:38625
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:38625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:09:58.652666 23689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:09:58.654047 23689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:09:58.665959 23697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:58.666019 23694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:09:58.665998 23695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:09:58.666607 23689 server_base.cc:1034] running on GCE node
I20260928 02:09:58.667438 23689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:09:58.668815 23689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:09:58.670053 23689 hybrid_clock.cc:648] HybridClock initialized: now 1790561398669993 us; error 49 us; skew 500 ppm
I20260928 02:09:58.670362 23689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:09:58 dist-test-slave-23bp krb5kdc[23676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561398, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:09:58.673734 23689 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:09:58.675383 23689 webserver.cc:460] Webserver started at http://127.1.182.190:39637/ using document root <none> and password file <none>
I20260928 02:09:58.676082 23689 fs_manager.cc:359] Metadata directory not provided
I20260928 02:09:58.676213 23689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:09:58.676541 23689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:09:58.679656 23689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "08653758503c409aadbe930fad032fcc"
format_stamp: "Formatted at 2026-09-28 02:09:58 on dist-test-slave-23bp"
server_key: "0f04b59c859426880f2d70213668b5a8"
server_key_iv: "330250d92bd826afd6a5ab0fa2d58b5e"
server_key_version: "encryptionkey@0"
I20260928 02:09:58.680465 23689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "08653758503c409aadbe930fad032fcc"
format_stamp: "Formatted at 2026-09-28 02:09:58 on dist-test-slave-23bp"
server_key: "0f04b59c859426880f2d70213668b5a8"
server_key_iv: "330250d92bd826afd6a5ab0fa2d58b5e"
server_key_version: "encryptionkey@0"
I20260928 02:09:58.685911 23689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 02:09:58.689486 23704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:09:58.690850 23689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:09:58.691028 23689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "08653758503c409aadbe930fad032fcc"
format_stamp: "Formatted at 2026-09-28 02:09:58 on dist-test-slave-23bp"
server_key: "0f04b59c859426880f2d70213668b5a8"
server_key_iv: "330250d92bd826afd6a5ab0fa2d58b5e"
server_key_version: "encryptionkey@0"
I20260928 02:09:58.691291 23689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:09:58.705093 23689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:09:58.709237 23689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:09:58.709525 23689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:09:58.728261 23689 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:38625
I20260928 02:09:58.728288 23756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:38625 every 8 connection(s)
I20260928 02:09:58.729661 23689 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:09:58.733355 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:09:58.736672 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 23689
I20260928 02:09:58.736851 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:09:58.737401 1754 external_mini_cluster.cc:1428] Setting key 252e9fb6afbe0ca225075a0b1c429f82
I20260928 02:09:58.745555 23757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc: Bootstrap starting.
I20260928 02:09:58.749158 23757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc: Neither blocks nor log segments found. Creating new log.
I20260928 02:09:58.750300 23757 log.cc:824] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:09:58.750469 23760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:58.739266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42995 (local address 127.1.182.190:38625)
0928 02:09:58.740028 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:58.740040 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:58.740083 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:09:58.741706 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:58.741728 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:58.742117 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:58.742425 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:58.742993 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:58.744343 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:58.746160 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:58.746802 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:58.746930 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:58.748292 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:58.748299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:58.748311 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:58.748819 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:58.748886 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:58.748909 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:58.749018 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:58.749374 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 02:09:58.749508 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":387,"threads_started":1}
I20260928 02:09:58.753199 23757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc: No bootstrap required, opened a new log
I20260928 02:09:58.754308 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:09:58.762209 23757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08653758503c409aadbe930fad032fcc" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38625 } }
I20260928 02:09:58.762629 23757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:09:58.762775 23757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08653758503c409aadbe930fad032fcc, State: Initialized, Role: FOLLOWER
I20260928 02:09:58.763441 23757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08653758503c409aadbe930fad032fcc" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38625 } }
I20260928 02:09:58.763675 23757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:09:58.763830 23757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:09:58.764005 23757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:09:58.766921 23760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:09:58.759517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53643 (local address 127.1.182.190:38625)
0928 02:09:58.759697 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:09:58.759704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:09:58.760279 (+ 575us) server_negotiation.cc:407] Connection header received
0928 02:09:58.760438 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:09:58.760445 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:09:58.760525 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:09:58.760652 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:09:58.761161 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:58.762255 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:58.764113 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:09:58.764850 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:09:58.764983 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:09:58.766145 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:09:58.766152 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:09:58.766156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:09:58.766177 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:09:58.766247 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:09:58.766273 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:09:58.766392 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:09:58.766548 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 02:09:58.766607 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:09:58.767374 23757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08653758503c409aadbe930fad032fcc" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38625 } }
I20260928 02:09:58.767988 23757 leader_election.cc:302] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08653758503c409aadbe930fad032fcc; no voters:
I20260928 02:09:58.768651 23757 leader_election.cc:288] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:09:58.768800 23767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:09:58.769550 23767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc [term 1 LEADER]: Becoming Leader. State: Replica: 08653758503c409aadbe930fad032fcc, State: Running, Role: LEADER
I20260928 02:09:58.770157 23767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08653758503c409aadbe930fad032fcc" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38625 } }
I20260928 02:09:58.771027 23757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:09:58.773804 23769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08653758503c409aadbe930fad032fcc. Latest consensus state: current_term: 1 leader_uuid: "08653758503c409aadbe930fad032fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08653758503c409aadbe930fad032fcc" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38625 } } }
I20260928 02:09:58.774240 23769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:58.773682 23768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08653758503c409aadbe930fad032fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08653758503c409aadbe930fad032fcc" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38625 } } }
I20260928 02:09:58.778718 23768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc [sys.catalog]: This master's current role is: LEADER
I20260928 02:09:58.783182 23774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:09:58.789023 23774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:09:58.798529 23774 catalog_manager.cc:1347] Generated new cluster ID: 813872f8d6814f64815d6c94cd326fb5
I20260928 02:09:58.798710 23774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:09:58.827225 23774 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:09:58.829128 23774 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:09:58.850080 23774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08653758503c409aadbe930fad032fcc: Generated new TSK 0
I20260928 02:09:58.850924 23774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:09:58.858361 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 23689
W20260928 02:09:58.859616 23752 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42995
W20260928 02:09:58.860436 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 23689: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:09:58.860594 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 23689
W20260928 02:09:58.877561 23674 connection.cc:535] client connection to 127.1.182.190:38625 recv error: Network error: failed to read from TLS socket (remote: 127.1.182.190:38625): Connection reset by peer (error 104)
2026-09-28T02:09:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2012 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:09:58 dist-test-slave-23bp krb5kdc[23788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:09:58 dist-test-slave-23bp krb5kdc[23788](info): set up 2 sockets
Sep 28 02:09:58 dist-test-slave-23bp krb5kdc[23788](info): commencing operation
krb5kdc: starting...
W20260928 02:10:00.022740 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:10:00 dist-test-slave-23bp krb5kdc[23788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:10:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:10:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:10:00.294706 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:36839
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:36597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:36839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:00.631198 23802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:00.631522 23802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:00.631695 23802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:00.639042 23802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:10:00.639213 23802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:00.639320 23802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:00.639426 23802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:10:00.639511 23802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:10:00.649528 23802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:36597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:36839
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:36839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:00.650926 23802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:00.652341 23802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:00.664556 23807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:00.664494 23808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:00.664788 23810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:00.665665 23802 server_base.cc:1034] running on GCE node
I20260928 02:10:00.666443 23802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:00.667943 23802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:00.669179 23802 hybrid_clock.cc:648] HybridClock initialized: now 1790561400669123 us; error 48 us; skew 500 ppm
I20260928 02:10:00.669622 23802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:00 dist-test-slave-23bp krb5kdc[23788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561400, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:00.679322 23802 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:10:00.680958 23802 webserver.cc:460] Webserver started at http://127.1.182.190:43151/ using document root <none> and password file <none>
I20260928 02:10:00.681643 23802 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:00.681744 23802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:00.682006 23802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:00.685101 23802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "eafcc8f146a24e54a56246432700b396"
format_stamp: "Formatted at 2026-09-28 02:10:00 on dist-test-slave-23bp"
server_key: "f357ca3b156ab1f5f42ff723c24dbc88"
server_key_iv: "f21d17d8db46e8dabc4a50ae6f6bc0b5"
server_key_version: "encryptionkey@0"
I20260928 02:10:00.685869 23802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "eafcc8f146a24e54a56246432700b396"
format_stamp: "Formatted at 2026-09-28 02:10:00 on dist-test-slave-23bp"
server_key: "f357ca3b156ab1f5f42ff723c24dbc88"
server_key_iv: "f21d17d8db46e8dabc4a50ae6f6bc0b5"
server_key_version: "encryptionkey@0"
I20260928 02:10:00.691331 23802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:10:00.694924 23817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:00.696545 23802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:10:00.696761 23802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "eafcc8f146a24e54a56246432700b396"
format_stamp: "Formatted at 2026-09-28 02:10:00 on dist-test-slave-23bp"
server_key: "f357ca3b156ab1f5f42ff723c24dbc88"
server_key_iv: "f21d17d8db46e8dabc4a50ae6f6bc0b5"
server_key_version: "encryptionkey@0"
I20260928 02:10:00.696980 23802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:00.702999 23802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:00.707530 23802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:00.707825 23802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:00.726704 23802 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:36839
I20260928 02:10:00.726738 23868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:36839 every 8 connection(s)
I20260928 02:10:00.728169 23802 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:10:00.730710 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 23802
I20260928 02:10:00.730921 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:10:00.731431 1754 external_mini_cluster.cc:1428] Setting key d97de0113f409bdfde05dd09e86796a2
I20260928 02:10:00.732031 23869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:00.740258 23872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:00.733233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52655 (local address 127.1.182.190:36839)
0928 02:10:00.734018 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:00.734030 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:00.734107 (+ 77us) server_negotiation.cc:407] Connection header received
0928 02:10:00.736745 (+ 2638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:00.736767 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:00.737115 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:00.737448 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:00.738522 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:00.738528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:00.738540 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:10:00.738553 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:00.738635 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:10:00.738655 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:00.738826 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:00.738935 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 02:10:00.739174 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":372,"threads_started":1}
I20260928 02:10:00.743947 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:10:00.745129 23869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396: Bootstrap starting.
I20260928 02:10:00.749343 23869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:00.750679 23869 log.cc:824] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:00.751897 23872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:00.749303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45567 (local address 127.1.182.190:36839)
0928 02:10:00.749478 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:00.749485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:00.749829 (+ 344us) server_negotiation.cc:407] Connection header received
0928 02:10:00.750011 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:00.750014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:00.750062 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:00.750139 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:00.751138 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:00.751142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:00.751145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:10:00.751159 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:00.751217 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:10:00.751245 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:00.751379 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:00.751503 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 02:10:00.751580 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:10:00.753968 23869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396: No bootstrap required, opened a new log
I20260928 02:10:00.762053 23869 raft_consensus.cc:348] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafcc8f146a24e54a56246432700b396" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36839 } }
I20260928 02:10:00.762403 23869 raft_consensus.cc:374] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:00.762475 23869 raft_consensus.cc:729] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafcc8f146a24e54a56246432700b396, State: Initialized, Role: FOLLOWER
I20260928 02:10:00.762990 23869 consensus_queue.cc:260] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafcc8f146a24e54a56246432700b396" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36839 } }
I20260928 02:10:00.763252 23869 raft_consensus.cc:388] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:10:00.763355 23869 raft_consensus.cc:482] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:10:00.763474 23869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:00.765635 23869 raft_consensus.cc:504] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafcc8f146a24e54a56246432700b396" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36839 } }
I20260928 02:10:00.766085 23869 leader_election.cc:302] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eafcc8f146a24e54a56246432700b396; no voters:
I20260928 02:10:00.766602 23869 leader_election.cc:288] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:10:00.766769 23880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:10:00.767490 23880 raft_consensus.cc:686] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396 [term 1 LEADER]: Becoming Leader. State: Replica: eafcc8f146a24e54a56246432700b396, State: Running, Role: LEADER
I20260928 02:10:00.768043 23880 consensus_queue.cc:237] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafcc8f146a24e54a56246432700b396" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36839 } }
I20260928 02:10:00.768955 23869 sys_catalog.cc:564] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:10:00.771246 23882 sys_catalog.cc:455] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eafcc8f146a24e54a56246432700b396. Latest consensus state: current_term: 1 leader_uuid: "eafcc8f146a24e54a56246432700b396" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafcc8f146a24e54a56246432700b396" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36839 } } }
I20260928 02:10:00.771200 23881 sys_catalog.cc:455] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eafcc8f146a24e54a56246432700b396" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafcc8f146a24e54a56246432700b396" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 36839 } } }
I20260928 02:10:00.771746 23882 sys_catalog.cc:458] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396 [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:00.771775 23881 sys_catalog.cc:458] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396 [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:00.777719 23888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:10:00.782838 23888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:10:00.791034 23888 catalog_manager.cc:1347] Generated new cluster ID: 38d1c8850db2481bbb6b9c2d02e134dc
I20260928 02:10:00.791162 23888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:10:00.800742 23888 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:10:00.801271 23888 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:10:00.810729 23888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eafcc8f146a24e54a56246432700b396: Generated new TSK 0
I20260928 02:10:00.811815 23888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:10:00.830905 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 23802
W20260928 02:10:00.832366 23864 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52655
W20260928 02:10:00.833119 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 23802: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:10:00.833241 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 23802
2026-09-28T02:10:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1969 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:10:00 dist-test-slave-23bp krb5kdc[23901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:10:00 dist-test-slave-23bp krb5kdc[23901](info): set up 2 sockets
Sep 28 02:10:00 dist-test-slave-23bp krb5kdc[23901](info): commencing operation
krb5kdc: starting...
W20260928 02:10:01.945822 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:10:02 dist-test-slave-23bp krb5kdc[23901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:10:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:10:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:10:02.206959 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:39601
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:35051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:39601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:02.542125 23914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:02.542459 23914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:02.542685 23914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:02.550741 23914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:10:02.550865 23914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:02.550974 23914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:02.551087 23914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:10:02.551159 23914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:10:02.561460 23914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:35051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:39601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:39601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:02.562855 23914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:02.564288 23914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:02.576745 23919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:02.576692 23920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:02.576766 23922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:02.577680 23914 server_base.cc:1034] running on GCE node
I20260928 02:10:02.578470 23914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:02.580030 23914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:02.581261 23914 hybrid_clock.cc:648] HybridClock initialized: now 1790561402581201 us; error 55 us; skew 500 ppm
I20260928 02:10:02.581653 23914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:02 dist-test-slave-23bp krb5kdc[23901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561402, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:02.585022 23914 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:10:02.586599 23914 webserver.cc:460] Webserver started at http://127.1.182.190:43113/ using document root <none> and password file <none>
I20260928 02:10:02.587417 23914 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:02.587579 23914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:02.587915 23914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:02.591274 23914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "938ed810e41c4085987a2babb5778be0"
format_stamp: "Formatted at 2026-09-28 02:10:02 on dist-test-slave-23bp"
server_key: "fabc312b4544735464c791915709ee79"
server_key_iv: "128a4f10409d6b3919fd8021703259da"
server_key_version: "encryptionkey@0"
I20260928 02:10:02.592145 23914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "938ed810e41c4085987a2babb5778be0"
format_stamp: "Formatted at 2026-09-28 02:10:02 on dist-test-slave-23bp"
server_key: "fabc312b4544735464c791915709ee79"
server_key_iv: "128a4f10409d6b3919fd8021703259da"
server_key_version: "encryptionkey@0"
I20260928 02:10:02.597757 23914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:10:02.601837 23929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:02.603480 23914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:10:02.603636 23914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "938ed810e41c4085987a2babb5778be0"
format_stamp: "Formatted at 2026-09-28 02:10:02 on dist-test-slave-23bp"
server_key: "fabc312b4544735464c791915709ee79"
server_key_iv: "128a4f10409d6b3919fd8021703259da"
server_key_version: "encryptionkey@0"
I20260928 02:10:02.603811 23914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:02.620317 23914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:02.624261 23914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:02.624507 23914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:02.643909 23914 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:39601
I20260928 02:10:02.643935 23980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:39601 every 8 connection(s)
I20260928 02:10:02.645421 23914 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:10:02.649441 23981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:02.651790 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 23914
I20260928 02:10:02.651983 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:10:02.652413 1754 external_mini_cluster.cc:1428] Setting key d0961b016f6e597e4eedbbbb7d23c453
I20260928 02:10:02.663509 23981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0: Bootstrap starting.
I20260928 02:10:02.667339 23981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:02.668483 23981 log.cc:824] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:10:02 dist-test-slave-23bp krb5kdc[23901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:10:02.671720 23981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0: No bootstrap required, opened a new log
I20260928 02:10:02.677497 23984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:02.654867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60115 (local address 127.1.182.190:39601)
0928 02:10:02.655927 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:02.655942 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:02.655994 (+ 52us) server_negotiation.cc:407] Connection header received
0928 02:10:02.658335 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:02.658364 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:02.658901 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:02.659379 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:02.660795 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:02.662781 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:02.664229 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:02.665324 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:02.665523 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:02.669946 (+ 4423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:02.669954 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:02.669969 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:02.670009 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:02.673173 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:02.674002 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:02.674009 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:02.674018 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:02.674216 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:02.674654 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:02.674657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:02.674659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:02.675339 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:02.675571 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:02.676010 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 02:10:02.676410 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":404,"threads_started":1}
I20260928 02:10:02.680691 23981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938ed810e41c4085987a2babb5778be0" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39601 } }
I20260928 02:10:02.681068 23981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:02.681195 23981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 938ed810e41c4085987a2babb5778be0, State: Initialized, Role: FOLLOWER
I20260928 02:10:02.681370 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:10:02.681842 23981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938ed810e41c4085987a2babb5778be0" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39601 } }
I20260928 02:10:02.682235 23981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:10:02.682374 23981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:10:02.682523 23981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:02.685940 23981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938ed810e41c4085987a2babb5778be0" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39601 } }
I20260928 02:10:02.686571 23981 leader_election.cc:302] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 938ed810e41c4085987a2babb5778be0; no voters:
I20260928 02:10:02.687305 23981 leader_election.cc:288] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:10:02.687428 23991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:10:02.689978 23981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:10:02.691376 23991 raft_consensus.cc:686] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0 [term 1 LEADER]: Becoming Leader. State: Replica: 938ed810e41c4085987a2babb5778be0, State: Running, Role: LEADER
I20260928 02:10:02.692195 23991 consensus_queue.cc:237] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938ed810e41c4085987a2babb5778be0" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39601 } }
I20260928 02:10:02.695573 23992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "938ed810e41c4085987a2babb5778be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938ed810e41c4085987a2babb5778be0" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39601 } } }
I20260928 02:10:02.695986 23992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:02.696523 23993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 938ed810e41c4085987a2babb5778be0. Latest consensus state: current_term: 1 leader_uuid: "938ed810e41c4085987a2babb5778be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938ed810e41c4085987a2babb5778be0" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 39601 } } }
I20260928 02:10:02.697321 23993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:02.710265 24000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:10:02.716303 24000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:10:02.718717 23984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:02.686622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51801 (local address 127.1.182.190:39601)
0928 02:10:02.686857 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:02.686868 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:02.693197 (+ 6329us) server_negotiation.cc:407] Connection header received
0928 02:10:02.693418 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:02.693424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:02.693503 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:02.693617 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:02.697594 (+ 3977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:02.699136 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:02.701564 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:02.702438 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:02.702545 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:02.706599 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:02.706609 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:02.706618 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:02.706673 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:02.712945 (+ 6272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:02.714134 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:02.714143 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:02.714148 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:02.714360 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:02.714964 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:02.714972 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:02.714978 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:02.715717 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:02.716136 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:02.717653 (+ 1517us) server_negotiation.cc:299] Negotiation successful
0928 02:10:02.717967 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:10:02.728878 24000 catalog_manager.cc:1347] Generated new cluster ID: 8ead820b654549a9b0abb12baefdeab9
I20260928 02:10:02.729013 24000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:10:02.745479 24000 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:10:02.746976 24000 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:10:02.757557 24000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 938ed810e41c4085987a2babb5778be0: Generated new TSK 0
I20260928 02:10:02.758412 24000 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:10:02.769044 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 23914
I20260928 02:10:02.770383 23976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:10:02.862275 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 23914
2026-09-28T02:10:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2033 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:10:02 dist-test-slave-23bp krb5kdc[24013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:10:02 dist-test-slave-23bp krb5kdc[24013](info): set up 2 sockets
Sep 28 02:10:02 dist-test-slave-23bp krb5kdc[24013](info): commencing operation
krb5kdc: starting...
W20260928 02:10:03.989886 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:10:04 dist-test-slave-23bp krb5kdc[24013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:10:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:10:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:10:04.282593 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:35335
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:39527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:35335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:04.616670 24027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:04.616984 24027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:04.617141 24027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:04.625748 24027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:10:04.625892 24027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:04.625976 24027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:04.626061 24027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:10:04.626142 24027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:10:04.636473 24027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:39527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:35335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:35335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:04.637882 24027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:04.639314 24027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:04.651422 24032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:04.651468 24033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:04.651494 24035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:04.652165 24027 server_base.cc:1034] running on GCE node
I20260928 02:10:04.652909 24027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:04.654386 24027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:04.655597 24027 hybrid_clock.cc:648] HybridClock initialized: now 1790561404655537 us; error 58 us; skew 500 ppm
I20260928 02:10:04.655936 24027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:04 dist-test-slave-23bp krb5kdc[24013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561404, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:04.659546 24027 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:10:04.661062 24027 webserver.cc:460] Webserver started at http://127.1.182.190:33205/ using document root <none> and password file <none>
I20260928 02:10:04.661783 24027 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:04.661935 24027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:04.662254 24027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:04.665406 24027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "9f690121c83e41138d84bec3c9a98dfb"
format_stamp: "Formatted at 2026-09-28 02:10:04 on dist-test-slave-23bp"
server_key: "a6dc49e00053df3e6168a39e8a93dd20"
server_key_iv: "480e353de981a6be5d6e0a1ecd9624e2"
server_key_version: "encryptionkey@0"
I20260928 02:10:04.666186 24027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "9f690121c83e41138d84bec3c9a98dfb"
format_stamp: "Formatted at 2026-09-28 02:10:04 on dist-test-slave-23bp"
server_key: "a6dc49e00053df3e6168a39e8a93dd20"
server_key_iv: "480e353de981a6be5d6e0a1ecd9624e2"
server_key_version: "encryptionkey@0"
I20260928 02:10:04.671656 24027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:10:04.675030 24042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:04.676421 24027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:10:04.676610 24027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "9f690121c83e41138d84bec3c9a98dfb"
format_stamp: "Formatted at 2026-09-28 02:10:04 on dist-test-slave-23bp"
server_key: "a6dc49e00053df3e6168a39e8a93dd20"
server_key_iv: "480e353de981a6be5d6e0a1ecd9624e2"
server_key_version: "encryptionkey@0"
I20260928 02:10:04.676815 24027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:04.692646 24027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:04.696799 24027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:04.697098 24027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:04.716791 24027 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:35335
I20260928 02:10:04.716811 24093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:35335 every 8 connection(s)
I20260928 02:10:04.718318 24027 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:10:04.718631 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 24027
I20260928 02:10:04.718809 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:10:04.719504 1754 external_mini_cluster.cc:1428] Setting key 8cf663ca2a79f5144b4289b4a0b9f70a
I20260928 02:10:04.723703 24094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:04.733484 24097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:04.721145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36913 (local address 127.1.182.190:35335)
0928 02:10:04.721962 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:04.721975 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:04.722017 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:10:04.723943 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:04.723967 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:04.724359 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:04.724693 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:04.725329 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:04.726723 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:04.728808 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:04.729547 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:04.729700 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:04.731042 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:04.731049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:04.731118 (+ 69us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:10:04.731701 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:04.731788 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:10:04.731824 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:04.731915 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:04.732167 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 02:10:04.732487 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":383,"threads_started":1}
I20260928 02:10:04.736972 24094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb: Bootstrap starting.
I20260928 02:10:04.737592 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:10:04.740236 24094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:04.741389 24094 log.cc:824] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:04.745155 24094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb: No bootstrap required, opened a new log
I20260928 02:10:04.748814 24097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:04.741990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49355 (local address 127.1.182.190:35335)
0928 02:10:04.742222 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:04.742232 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:04.742805 (+ 573us) server_negotiation.cc:407] Connection header received
0928 02:10:04.742959 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:04.742966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:04.743061 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:04.743187 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:04.743786 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:04.744498 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:04.746300 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:04.746759 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:04.746859 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:04.748090 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:04.748095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:04.748098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:10:04.748117 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:04.748155 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:10:04.748172 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:04.748249 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:04.748399 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 02:10:04.748469 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:10:04.753940 24094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f690121c83e41138d84bec3c9a98dfb" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 35335 } }
I20260928 02:10:04.754297 24094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:04.754374 24094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f690121c83e41138d84bec3c9a98dfb, State: Initialized, Role: FOLLOWER
I20260928 02:10:04.754957 24094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f690121c83e41138d84bec3c9a98dfb" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 35335 } }
I20260928 02:10:04.755255 24094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:10:04.755362 24094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:10:04.755486 24094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:04.757751 24094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f690121c83e41138d84bec3c9a98dfb" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 35335 } }
I20260928 02:10:04.758343 24094 leader_election.cc:302] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f690121c83e41138d84bec3c9a98dfb; no voters:
I20260928 02:10:04.758951 24094 leader_election.cc:288] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:10:04.759181 24104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:10:04.759907 24104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb [term 1 LEADER]: Becoming Leader. State: Replica: 9f690121c83e41138d84bec3c9a98dfb, State: Running, Role: LEADER
I20260928 02:10:04.760486 24104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f690121c83e41138d84bec3c9a98dfb" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 35335 } }
I20260928 02:10:04.761209 24094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:10:04.763522 24106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f690121c83e41138d84bec3c9a98dfb. Latest consensus state: current_term: 1 leader_uuid: "9f690121c83e41138d84bec3c9a98dfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f690121c83e41138d84bec3c9a98dfb" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 35335 } } }
I20260928 02:10:04.763499 24105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f690121c83e41138d84bec3c9a98dfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f690121c83e41138d84bec3c9a98dfb" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 35335 } } }
I20260928 02:10:04.764159 24105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:04.764048 24106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:04.766764 24110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:10:04.773469 24110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:10:04.782848 24110 catalog_manager.cc:1347] Generated new cluster ID: 227aaa7af92c4117ae60014335750b97
I20260928 02:10:04.782974 24110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:10:04.799113 24110 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:10:04.800601 24110 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:10:04.810654 24110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f690121c83e41138d84bec3c9a98dfb: Generated new TSK 0
I20260928 02:10:04.811682 24110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:10:04.827039 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 24027
W20260928 02:10:04.828137 24089 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36913
W20260928 02:10:04.828843 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 24027: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:10:04.828979 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 24027
2026-09-28T02:10:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1965 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:10:04 dist-test-slave-23bp krb5kdc[24125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:10:04 dist-test-slave-23bp krb5kdc[24125](info): set up 2 sockets
Sep 28 02:10:04 dist-test-slave-23bp krb5kdc[24125](info): commencing operation
krb5kdc: starting...
W20260928 02:10:05.962318 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:10:06 dist-test-slave-23bp krb5kdc[24125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:10:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:10:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:10:06.239389 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:41413
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:40759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:41413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:06.594643 24138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:06.594964 24138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:06.595084 24138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:06.602612 24138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:10:06.602797 24138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:06.602922 24138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:06.603035 24138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:10:06.603138 24138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:10:06.613684 24138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:40759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:41413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:41413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:06.615203 24138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:06.616633 24138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:06.629573 24146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:06.629619 24144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:06.629817 24143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:06.630870 24138 server_base.cc:1034] running on GCE node
I20260928 02:10:06.632174 24138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:06.634081 24138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:06.635319 24138 hybrid_clock.cc:648] HybridClock initialized: now 1790561406635270 us; error 48 us; skew 500 ppm
I20260928 02:10:06.635730 24138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:06 dist-test-slave-23bp krb5kdc[24125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561406, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:06.642611 24138 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:10:06.644315 24138 webserver.cc:460] Webserver started at http://127.1.182.190:46351/ using document root <none> and password file <none>
I20260928 02:10:06.645051 24138 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:06.645213 24138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:06.645534 24138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:06.648742 24138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "975c2f86cdfa4d6d8e6d78c78ec67350"
format_stamp: "Formatted at 2026-09-28 02:10:06 on dist-test-slave-23bp"
server_key: "3d6f150fb8e05f292b149bd77f5008b3"
server_key_iv: "ab619c3ba33da5a6f70dcd8c2e833fe2"
server_key_version: "encryptionkey@0"
I20260928 02:10:06.649550 24138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "975c2f86cdfa4d6d8e6d78c78ec67350"
format_stamp: "Formatted at 2026-09-28 02:10:06 on dist-test-slave-23bp"
server_key: "3d6f150fb8e05f292b149bd77f5008b3"
server_key_iv: "ab619c3ba33da5a6f70dcd8c2e833fe2"
server_key_version: "encryptionkey@0"
I20260928 02:10:06.655010 24138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:10:06.658990 24153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:06.660501 24138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:10:06.660703 24138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "975c2f86cdfa4d6d8e6d78c78ec67350"
format_stamp: "Formatted at 2026-09-28 02:10:06 on dist-test-slave-23bp"
server_key: "3d6f150fb8e05f292b149bd77f5008b3"
server_key_iv: "ab619c3ba33da5a6f70dcd8c2e833fe2"
server_key_version: "encryptionkey@0"
I20260928 02:10:06.660924 24138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:06.666770 24138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:06.671164 24138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:06.671478 24138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:06.690534 24138 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:41413
I20260928 02:10:06.690600 24205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:41413 every 8 connection(s)
I20260928 02:10:06.692108 24138 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:10:06.696166 24206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:06.699947 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 24138
I20260928 02:10:06.700132 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:10:06.700687 1754 external_mini_cluster.cc:1428] Setting key 17453f2592ca7503013eb1fd557a2299
I20260928 02:10:06.709029 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:06.702728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60977 (local address 127.1.182.190:41413)
0928 02:10:06.703585 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:06.703598 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:06.703642 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:10:06.705338 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:06.705362 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:06.705763 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:06.706107 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:06.707280 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:06.707286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:06.707303 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:10:06.707317 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:06.707380 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:10:06.707400 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:06.707586 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:06.707762 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 02:10:06.708077 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":410,"threads_started":1}
I20260928 02:10:06.709545 24206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350: Bootstrap starting.
I20260928 02:10:06.712864 24206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:06.713045 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:10:06.714139 24206 log.cc:824] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:06.717711 24206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350: No bootstrap required, opened a new log
I20260928 02:10:06.720927 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:06.718284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37533 (local address 127.1.182.190:41413)
0928 02:10:06.718500 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:06.718506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:06.718886 (+ 380us) server_negotiation.cc:407] Connection header received
0928 02:10:06.719060 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:06.719111 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:06.719171 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:06.719256 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:06.720232 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:06.720237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:06.720240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:10:06.720255 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:06.720301 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:10:06.720318 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:06.720424 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:06.720560 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 02:10:06.720607 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:10:06.726275 24206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975c2f86cdfa4d6d8e6d78c78ec67350" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 41413 } }
I20260928 02:10:06.726670 24206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:06.726781 24206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975c2f86cdfa4d6d8e6d78c78ec67350, State: Initialized, Role: FOLLOWER
I20260928 02:10:06.727450 24206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975c2f86cdfa4d6d8e6d78c78ec67350" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 41413 } }
I20260928 02:10:06.727698 24206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:10:06.727855 24206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:10:06.728068 24206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:06.730312 24206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975c2f86cdfa4d6d8e6d78c78ec67350" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 41413 } }
I20260928 02:10:06.730834 24206 leader_election.cc:302] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 975c2f86cdfa4d6d8e6d78c78ec67350; no voters:
I20260928 02:10:06.731534 24206 leader_election.cc:288] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:10:06.731706 24216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:10:06.732983 24216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350 [term 1 LEADER]: Becoming Leader. State: Replica: 975c2f86cdfa4d6d8e6d78c78ec67350, State: Running, Role: LEADER
I20260928 02:10:06.733703 24216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975c2f86cdfa4d6d8e6d78c78ec67350" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 41413 } }
I20260928 02:10:06.733881 24206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:10:06.736312 24218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 975c2f86cdfa4d6d8e6d78c78ec67350. Latest consensus state: current_term: 1 leader_uuid: "975c2f86cdfa4d6d8e6d78c78ec67350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975c2f86cdfa4d6d8e6d78c78ec67350" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 41413 } } }
I20260928 02:10:06.736688 24218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350 [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:06.736989 24217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "975c2f86cdfa4d6d8e6d78c78ec67350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975c2f86cdfa4d6d8e6d78c78ec67350" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 41413 } } }
I20260928 02:10:06.737339 24217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350 [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:06.739452 24223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:10:06.745203 24223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:10:06.754701 24223 catalog_manager.cc:1347] Generated new cluster ID: 2c1b9e84b05d43af9d4a65fd6c600d8d
I20260928 02:10:06.754904 24223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:10:06.777273 24223 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:10:06.777725 24223 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:10:06.788007 24223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 975c2f86cdfa4d6d8e6d78c78ec67350: Generated new TSK 0
I20260928 02:10:06.788882 24223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:10:06.799001 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 24138
W20260928 02:10:06.800612 24201 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60977
W20260928 02:10:06.801317 1754 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 24138: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:10:06.801488 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 24138
2026-09-28T02:10:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1972 ms)
[----------] 18 tests from AuthTokenIssuingTest (35760 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:10:06 dist-test-slave-23bp krb5kdc[24237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:10:06 dist-test-slave-23bp krb5kdc[24237](info): set up 2 sockets
Sep 28 02:10:06 dist-test-slave-23bp krb5kdc[24237](info): commencing operation
krb5kdc: starting...
W20260928 02:10:07.926095 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:10:08 dist-test-slave-23bp krb5kdc[24237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:10:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:10:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:10:08.198704 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:44847
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:46367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:44847,127.1.182.189:38497,127.1.182.188:45333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:08.528916 24250 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:10:08.529328 24250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:08.529459 24250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:08.529598 24250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:08.537206 24250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:10:08.537344 24250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:08.537477 24250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:08.537564 24250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:10:08.537680 24250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:10:08.547951 24250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:46367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:44847,127.1.182.189:38497,127.1.182.188:45333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:44847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:08.549886 24250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:08.551304 24250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:08.563831 24256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:08.563870 24258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:08.564143 24255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:08.565178 24250 server_base.cc:1034] running on GCE node
I20260928 02:10:08.565990 24250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:08.567632 24250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:08.568897 24250 hybrid_clock.cc:648] HybridClock initialized: now 1790561408568826 us; error 61 us; skew 500 ppm
I20260928 02:10:08.569250 24250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:08 dist-test-slave-23bp krb5kdc[24237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561408, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:08.572609 24250 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:10:08.574270 24250 webserver.cc:460] Webserver started at http://127.1.182.190:43963/ using document root <none> and password file <none>
I20260928 02:10:08.574996 24250 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:08.575158 24250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:08.575488 24250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:08.579303 24250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9"
format_stamp: "Formatted at 2026-09-28 02:10:08 on dist-test-slave-23bp"
server_key: "802071f81e470ff7c5d11e1f09c9076b"
server_key_iv: "369eeb8bea9ffe9abec7049e45a30021"
server_key_version: "encryptionkey@0"
I20260928 02:10:08.580135 24250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9"
format_stamp: "Formatted at 2026-09-28 02:10:08 on dist-test-slave-23bp"
server_key: "802071f81e470ff7c5d11e1f09c9076b"
server_key_iv: "369eeb8bea9ffe9abec7049e45a30021"
server_key_version: "encryptionkey@0"
I20260928 02:10:08.585479 24250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:10:08.589076 24265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:08.590514 24250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:10:08.590704 24250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9"
format_stamp: "Formatted at 2026-09-28 02:10:08 on dist-test-slave-23bp"
server_key: "802071f81e470ff7c5d11e1f09c9076b"
server_key_iv: "369eeb8bea9ffe9abec7049e45a30021"
server_key_version: "encryptionkey@0"
I20260928 02:10:08.590931 24250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:08.614403 24250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:08.618722 24250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:08.619035 24250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:08.638270 24250 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:44847
I20260928 02:10:08.638276 24316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:44847 every 8 connection(s)
I20260928 02:10:08.640153 24250 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:10:08.643844 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 24250
I20260928 02:10:08.644061 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:10:08.644505 24317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:08.644738 1754 external_mini_cluster.cc:1428] Setting key aa0a5bd2346d25ddeffb343523e32d41
I20260928 02:10:08.651051 24317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44847 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:10:08 dist-test-slave-23bp krb5kdc[24237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561408, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.182.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.189@KRBTEST.COM" created.
I20260928 02:10:08.674856 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:08.654395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:44847 (local address 127.0.0.1:43372)
0928 02:10:08.655318 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:08.655358 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:08.657483 (+ 2125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:08.658324 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:08.658331 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:08.659054 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:08.659472 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:08.659490 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:08.661043 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:08.661053 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:08.662130 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:08.662142 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:08.663175 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:08.663182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:08.663369 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:08.664085 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:08.664108 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:08.666696 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:08.670242 (+ 3546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:08.670253 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:08.670262 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:08.670827 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:08.671325 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:08.671332 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:08.671335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:08.671489 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:08.672301 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:08.672314 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:08.672521 (+ 207us) client_negotiation.cc:769] Sending connection context
0928 02:10:08.672775 (+ 254us) client_negotiation.cc:240] Negotiation successful
0928 02:10:08.673439 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":180,"thread_start_us":363,"threads_started":1}
I20260928 02:10:08.674898 24321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:08.655782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43372 (local address 127.1.182.190:44847)
0928 02:10:08.656476 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:08.656486 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:08.656546 (+ 60us) server_negotiation.cc:407] Connection header received
0928 02:10:08.657863 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:08.657868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:08.658101 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:08.658318 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:08.659689 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:08.660861 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:08.662275 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:08.662836 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:08.663151 (+ 315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:08.666968 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:08.666976 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:08.666989 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:08.667021 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:08.670071 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:08.670981 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:08.670988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:08.670990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:08.671215 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:08.671613 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:08.671618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:08.671621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:08.672153 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:08.672363 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:08.672900 (+ 537us) server_negotiation.cc:299] Negotiation successful
0928 02:10:08.673322 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":478,"thread_start_us":301,"threads_started":1}
I20260928 02:10:08.678210 24317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:10:08.679388 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:08.678953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38497 (local address 127.0.0.1:48670)
0928 02:10:08.679177 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:08.679277 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 02:10:08.679769 24268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38497: connect: Connection refused (error 111)
I20260928 02:10:08.681453 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:08.680993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38497 (local address 127.0.0.1:48680)
0928 02:10:08.681170 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:08.681266 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 02:10:08.681668 24317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.189:38497: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38497: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.189.keytab.
Entry for principal kudu/127.1.182.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.189.keytab.
I20260928 02:10:08.702399 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.189:38497
--webserver_interface=127.1.182.189
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.189.keytab
--principal=kudu/127.1.182.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:46367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:44847,127.1.182.189:38497,127.1.182.188:45333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:10:08.731055 24317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } attempt: 1
I20260928 02:10:08.732303 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:08.731779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38497 (local address 127.0.0.1:48684)
0928 02:10:08.732001 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:08.732144 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 02:10:08.733364 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:08.732969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38497 (local address 127.0.0.1:48690)
0928 02:10:08.733118 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:08.733202 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 02:10:08.733557 24317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.189:38497: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38497: connect: Connection refused (error 111)
I20260928 02:10:08.801838 24317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } attempt: 2
I20260928 02:10:08.803231 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:08.802603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38497 (local address 127.0.0.1:48700)
0928 02:10:08.802888 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:08.803010 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 02:10:08.804278 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:08.803881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38497 (local address 127.0.0.1:48710)
0928 02:10:08.804045 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:08.804126 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 02:10:08.804495 24317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.189:38497: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38497: connect: Connection refused (error 111)
I20260928 02:10:08.895781 24317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } attempt: 3
I20260928 02:10:08.897059 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:08.896451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38497 (local address 127.0.0.1:48718)
0928 02:10:08.896722 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:08.896862 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 02:10:08.898335 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:08.897832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38497 (local address 127.0.0.1:48726)
0928 02:10:08.898023 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:08.898131 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 02:10:08.898593 24317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.189:38497: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38497: connect: Connection refused (error 111)
W20260928 02:10:09.036679 24324 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:10:09.037072 24324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:09.037197 24324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:09.037338 24324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 02:10:09.041850 24317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } attempt: 4
I20260928 02:10:09.043138 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.042486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38497 (local address 127.0.0.1:48734)
0928 02:10:09.042835 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.042945 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260928 02:10:09.044214 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.043819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38497 (local address 127.0.0.1:48746)
0928 02:10:09.043967 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.044065 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 02:10:09.044411 24317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.189:38497: Network error: Client connection negotiation failed: client connection to 127.1.182.189:38497: connect: Connection refused (error 111)
W20260928 02:10:09.045612 24324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:10:09.045689 24324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:09.045810 24324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:09.045890 24324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:10:09.045986 24324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:10:09.056428 24324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:46367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:44847,127.1.182.189:38497,127.1.182.188:45333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.189.keytab
--principal=kudu/127.1.182.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.189:38497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.182.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:09.058020 24324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:09.059544 24324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:09.072103 24332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:09.072176 24330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:09.072299 24329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:09.072867 24324 server_base.cc:1034] running on GCE node
I20260928 02:10:09.073642 24324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:09.075137 24324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:09.076380 24324 hybrid_clock.cc:648] HybridClock initialized: now 1790561409076290 us; error 66 us; skew 500 ppm
I20260928 02:10:09.076717 24324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:09 dist-test-slave-23bp krb5kdc[24237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:09.079944 24324 init.cc:377] Logged in from keytab as kudu/127.1.182.189@KRBTEST.COM (short username kudu)
I20260928 02:10:09.081385 24324 webserver.cc:460] Webserver started at http://127.1.182.189:40597/ using document root <none> and password file <none>
I20260928 02:10:09.082077 24324 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:09.082211 24324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:09.082547 24324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:09.085592 24324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-1/data/instance:
uuid: "3a1d9f1f87d24dfb9085919adb9b820e"
format_stamp: "Formatted at 2026-09-28 02:10:09 on dist-test-slave-23bp"
server_key: "a7d6bed1bae6a5713baee9737ed68286"
server_key_iv: "95f007d24a34173e54f95dd2a188a417"
server_key_version: "encryptionkey@0"
I20260928 02:10:09.086403 24324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-1/wal/instance:
uuid: "3a1d9f1f87d24dfb9085919adb9b820e"
format_stamp: "Formatted at 2026-09-28 02:10:09 on dist-test-slave-23bp"
server_key: "a7d6bed1bae6a5713baee9737ed68286"
server_key_iv: "95f007d24a34173e54f95dd2a188a417"
server_key_version: "encryptionkey@0"
I20260928 02:10:09.091795 24324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:10:09.095594 24339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:09.097060 24324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 02:10:09.097244 24324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-1/wal
uuid: "3a1d9f1f87d24dfb9085919adb9b820e"
format_stamp: "Formatted at 2026-09-28 02:10:09 on dist-test-slave-23bp"
server_key: "a7d6bed1bae6a5713baee9737ed68286"
server_key_iv: "95f007d24a34173e54f95dd2a188a417"
server_key_version: "encryptionkey@0"
I20260928 02:10:09.097476 24324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:09.112197 24324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:09.117077 24324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:09.117376 24324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:09.136934 24391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.189:38497 every 8 connection(s)
I20260928 02:10:09.136941 24324 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.189:38497
I20260928 02:10:09.138448 24324 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-1/data/info.pb
I20260928 02:10:09.142354 24392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:09.146867 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 24324
I20260928 02:10:09.147034 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-1/wal/instance
I20260928 02:10:09.147459 1754 external_mini_cluster.cc:1428] Setting key 8dfc94fb90cc8f5b1184c35954fca8ac
I20260928 02:10:09.149055 24392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44847 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:10:09 dist-test-slave-23bp krb5kdc[24237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.189@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:10:09.172708 24321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.151807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43380 (local address 127.1.182.190:44847)
0928 02:10:09.152054 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:09.152062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:09.153214 (+ 1152us) server_negotiation.cc:407] Connection header received
0928 02:10:09.155184 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:09.155193 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:09.155295 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:09.155423 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:09.157483 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.158300 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.160776 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.161202 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.161300 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.166015 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:09.166026 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:09.166031 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:09.166067 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:09.169056 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.170129 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.170133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.170135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.170264 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.170812 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.170816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.170818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.171218 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:09.171336 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:09.172294 (+ 958us) server_negotiation.cc:299] Negotiation successful
0928 02:10:09.172471 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
WARNING: no policy specified for kudu/127.1.182.188@KRBTEST.COM; defaulting to no policy
I20260928 02:10:09.174137 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.152226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:44847 (local address 127.0.0.1:43380)
0928 02:10:09.153023 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.153076 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:09.154849 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:09.155727 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:09.155735 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:09.156472 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:09.157277 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:09.157304 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.158486 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.158501 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:09.160603 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:09.160621 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.161462 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.161476 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:09.161827 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.162567 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:09.162602 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:09.165594 (+ 2992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:09.169210 (+ 3616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:09.169222 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:09.169243 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:09.169986 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:09.170384 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:09.170391 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:09.170397 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:09.170663 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:09.171362 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:09.171372 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:09.171874 (+ 502us) client_negotiation.cc:769] Sending connection context
0928 02:10:09.172216 (+ 342us) client_negotiation.cc:240] Negotiation successful
0928 02:10:09.172834 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":351,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.188@KRBTEST.COM" created.
I20260928 02:10:09.176117 24392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:10:09 dist-test-slave-23bp krb5kdc[24237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.189@KRBTEST.COM for kudu/127.1.182.189@KRBTEST.COM
I20260928 02:10:09.194073 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.176997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38497 (local address 127.0.0.1:48762)
0928 02:10:09.177163 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.177178 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:09.177264 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:09.178505 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:09.178512 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:09.179043 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:09.179280 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:09.179295 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.181108 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.181116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:09.182660 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:09.182677 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.183801 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.183809 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:09.184044 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.184714 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:09.184741 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:09.186909 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:09.191478 (+ 4569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:09.191482 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:09.191485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:09.191862 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:09.192291 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:09.192294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:09.192297 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:09.192454 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:09.193228 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:09.193232 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:09.193351 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 02:10:09.193487 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 02:10:09.193765 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 02:10:09.194494 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.177276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48762 (local address 127.1.182.189:38497)
0928 02:10:09.177974 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:09.177984 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:09.178053 (+ 69us) server_negotiation.cc:407] Connection header received
0928 02:10:09.178107 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:09.178117 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:09.178327 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:09.178467 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:09.179452 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.180939 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.182799 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.183644 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.183769 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.187161 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:09.187178 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:09.187198 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:09.187240 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:09.191336 (+ 4096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.192015 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.192027 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.192033 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.192176 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.192569 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.192577 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.192583 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.193091 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:09.193300 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:09.193660 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 02:10:09.193949 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":345,"threads_started":1}
I20260928 02:10:09.197167 24392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:10:09.198284 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.197810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45333 (local address 127.0.0.1:46614)
0928 02:10:09.198015 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.198128 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 02:10:09.198541 24342 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
I20260928 02:10:09.200233 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.199884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45333 (local address 127.0.0.1:46624)
0928 02:10:09.200039 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.200126 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 02:10:09.200450 24392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:45333: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.188.keytab.
Entry for principal kudu/127.1.182.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.188.keytab.
I20260928 02:10:09.210955 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.188:45333
--webserver_interface=127.1.182.188
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.188.keytab
--principal=kudu/127.1.182.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:46367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:44847,127.1.182.189:38497,127.1.182.188:45333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:10:09.249809 24392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } attempt: 1
I20260928 02:10:09.251135 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.250534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45333 (local address 127.0.0.1:46632)
0928 02:10:09.250780 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.250912 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 02:10:09.252341 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.251878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45333 (local address 127.0.0.1:46642)
0928 02:10:09.252108 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.252207 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 02:10:09.252570 24392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:45333: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
I20260928 02:10:09.320842 24392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } attempt: 2
I20260928 02:10:09.322373 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.321654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45333 (local address 127.0.0.1:46648)
0928 02:10:09.321990 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.322142 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 02:10:09.323680 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.323270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45333 (local address 127.0.0.1:46650)
0928 02:10:09.323427 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.323524 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 02:10:09.323909 24392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:45333: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
I20260928 02:10:09.343645 24317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } attempt: 5
Sep 28 02:10:09 dist-test-slave-23bp krb5kdc[24237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561408, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for kudu/127.1.182.189@KRBTEST.COM
I20260928 02:10:09.359233 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.344415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38497 (local address 127.0.0.1:48768)
0928 02:10:09.344640 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.344664 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:09.344798 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:09.345265 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:09.345268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:09.345625 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:09.345804 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:09.345812 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.347377 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.347382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:09.348303 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:09.348314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.349360 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.349365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:09.349577 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.350310 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:09.350329 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:09.352742 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:09.356710 (+ 3968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:09.356714 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:09.356717 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:09.357091 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:09.357484 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:09.357487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:09.357490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:09.357619 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:09.358269 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:09.358272 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:09.358439 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 02:10:09.358664 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 02:10:09.358911 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 02:10:09.359449 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.344624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48768 (local address 127.1.182.189:38497)
0928 02:10:09.344829 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:09.344836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:09.344882 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:10:09.344949 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:09.344955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:09.345118 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:09.345229 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:09.345975 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.347247 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.348461 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.349156 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.349286 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.352963 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:09.352970 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:09.352974 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:09.353024 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:09.356541 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.357219 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.357224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.357228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.357375 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.357724 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.357729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.357732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.358142 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:09.358317 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:09.358787 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 02:10:09.359036 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 02:10:09.360605 24317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:10:09.361632 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.361262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45333 (local address 127.0.0.1:46666)
0928 02:10:09.361424 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.361497 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 02:10:09.362520 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.362210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45333 (local address 127.0.0.1:46676)
0928 02:10:09.362350 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.362431 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:10:09.362743 24317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:45333: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
I20260928 02:10:09.414040 24317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } attempt: 1
I20260928 02:10:09.415163 24392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } attempt: 3
I20260928 02:10:09.415529 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.414844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45333 (local address 127.0.0.1:46682)
0928 02:10:09.415183 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.415279 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260928 02:10:09.416342 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.415777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45333 (local address 127.0.0.1:46690)
0928 02:10:09.416017 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.416158 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260928 02:10:09.416659 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.416257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45333 (local address 127.0.0.1:46698)
0928 02:10:09.416436 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.416528 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 02:10:09.417004 24317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:45333: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
I20260928 02:10:09.417779 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.417201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45333 (local address 127.0.0.1:46700)
0928 02:10:09.417501 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.417611 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
W20260928 02:10:09.418092 24392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:45333: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
I20260928 02:10:09.485320 24317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } attempt: 2
I20260928 02:10:09.486617 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.485987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45333 (local address 127.0.0.1:46712)
0928 02:10:09.486259 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.486390 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 02:10:09.487838 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.487394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45333 (local address 127.0.0.1:46722)
0928 02:10:09.487602 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.487696 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 02:10:09.488081 24317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:45333: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
W20260928 02:10:09.552516 24399 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:10:09.552846 24399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:09.552908 24399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:09.552990 24399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:09.560264 24399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:10:09.560355 24399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:09.560413 24399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:09.560456 24399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:10:09.560518 24399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:10:09.561345 24392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } attempt: 4
I20260928 02:10:09.562582 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.562002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45333 (local address 127.0.0.1:46726)
0928 02:10:09.562254 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.562342 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 02:10:09.563621 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.563287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45333 (local address 127.0.0.1:46732)
0928 02:10:09.563434 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.563511 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 02:10:09.563874 24392 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:45333: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
I20260928 02:10:09.571166 24399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:46367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:44847,127.1.182.189:38497,127.1.182.188:45333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.188.keytab
--principal=kudu/127.1.182.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.188:45333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.182.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:09.572556 24399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:09.573905 24399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:09.585728 24404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:09.585752 24405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:09.585685 24407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:09.586553 24399 server_base.cc:1034] running on GCE node
I20260928 02:10:09.587554 24399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:09.589028 24399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:09.590255 24399 hybrid_clock.cc:648] HybridClock initialized: now 1790561409590169 us; error 67 us; skew 500 ppm
I20260928 02:10:09.590596 24399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:09 dist-test-slave-23bp krb5kdc[24237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:09.593979 24399 init.cc:377] Logged in from keytab as kudu/127.1.182.188@KRBTEST.COM (short username kudu)
I20260928 02:10:09.594314 24317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } attempt: 3
I20260928 02:10:09.595568 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.595137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45333 (local address 127.0.0.1:46742)
0928 02:10:09.595313 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.595396 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 02:10:09.595965 24399 webserver.cc:460] Webserver started at http://127.1.182.188:36927/ using document root <none> and password file <none>
I20260928 02:10:09.596627 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.596306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45333 (local address 127.0.0.1:46754)
0928 02:10:09.596432 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.596503 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 02:10:09.596835 24399 fs_manager.cc:359] Metadata directory not provided
W20260928 02:10:09.596894 24317 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:45333: Network error: Client connection negotiation failed: client connection to 127.1.182.188:45333: connect: Connection refused (error 111)
I20260928 02:10:09.596983 24399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:09.597334 24399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:09.600394 24399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-2/data/instance:
uuid: "0876a5c5647442f49ca9f06e28859958"
format_stamp: "Formatted at 2026-09-28 02:10:09 on dist-test-slave-23bp"
server_key: "928ffb7296b9d06a5272c009097f2a1d"
server_key_iv: "2c37dbfdd3b2c2bdb05790a8ac621c21"
server_key_version: "encryptionkey@0"
I20260928 02:10:09.601240 24399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-2/wal/instance:
uuid: "0876a5c5647442f49ca9f06e28859958"
format_stamp: "Formatted at 2026-09-28 02:10:09 on dist-test-slave-23bp"
server_key: "928ffb7296b9d06a5272c009097f2a1d"
server_key_iv: "2c37dbfdd3b2c2bdb05790a8ac621c21"
server_key_version: "encryptionkey@0"
I20260928 02:10:09.606341 24399 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:10:09.609733 24414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:09.611006 24399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:10:09.611258 24399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-2/wal
uuid: "0876a5c5647442f49ca9f06e28859958"
format_stamp: "Formatted at 2026-09-28 02:10:09 on dist-test-slave-23bp"
server_key: "928ffb7296b9d06a5272c009097f2a1d"
server_key_iv: "2c37dbfdd3b2c2bdb05790a8ac621c21"
server_key_version: "encryptionkey@0"
I20260928 02:10:09.611472 24399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:09.630493 24399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:09.635136 24399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:09.635438 24399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:09.656499 24399 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.188:45333
I20260928 02:10:09.656534 24465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.188:45333 every 8 connection(s)
I20260928 02:10:09.657866 24399 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-2/data/info.pb
I20260928 02:10:09.661649 24466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:09.666373 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 24399
I20260928 02:10:09.666566 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561052730740-1754-0/minicluster-data/master-2/wal/instance
I20260928 02:10:09.667145 1754 external_mini_cluster.cc:1428] Setting key b8a5d158bc93fa407858ea2323550037
I20260928 02:10:09.669242 24466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44847 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:10:09 dist-test-slave-23bp krb5kdc[24237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.188@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
Sep 28 02:10:09 dist-test-slave-23bp krb5kdc[24237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.188@KRBTEST.COM
I20260928 02:10:09.687729 24321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.672144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43390 (local address 127.1.182.190:44847)
0928 02:10:09.672352 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:09.672359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:09.672732 (+ 373us) server_negotiation.cc:407] Connection header received
0928 02:10:09.672945 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:09.672950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:09.673040 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:09.673151 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:09.674358 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.675203 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.676655 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.677088 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.677231 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.680939 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:09.680944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:09.680947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:09.681037 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:09.684453 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.685271 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.685285 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.685289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.685444 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.685912 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.685918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.685923 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.686222 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:09.686397 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:09.687036 (+ 639us) server_negotiation.cc:299] Negotiation successful
0928 02:10:09.687308 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 02:10:09.688777 24469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.669101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.1.182.188:45333)
0928 02:10:09.669925 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:09.669937 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:09.669978 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:10:09.672095 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:09.672134 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:09.672669 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:09.673003 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:09.674478 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.675767 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.677031 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.677561 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.677700 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.681982 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:09.681991 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:09.682003 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:09.682031 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:09.684883 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.685530 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.685540 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.685544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.685701 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.686081 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.686086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.686091 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.686602 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:09.686923 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:09.687274 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 02:10:09.687643 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":605,"thread_start_us":396,"threads_started":1}
I20260928 02:10:09.688825 24470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.671892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:44847 (local address 127.0.0.1:43390)
0928 02:10:09.672595 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.672629 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:09.672817 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:09.673214 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:09.673221 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:09.673839 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:09.674192 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:09.674209 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.675355 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.675364 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:09.676512 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:09.676524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.677338 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.677345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:09.677585 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.678291 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:09.678312 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:09.680735 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:09.684614 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:09.684627 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:09.684641 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:09.685127 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:09.685565 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:09.685570 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:09.685575 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:09.685778 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:09.686366 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:09.686385 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:09.686621 (+ 236us) client_negotiation.cc:769] Sending connection context
0928 02:10:09.686928 (+ 307us) client_negotiation.cc:240] Negotiation successful
0928 02:10:09.687504 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":1,"thread_start_us":298,"threads_started":1}
I20260928 02:10:09.691305 24466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:10:09 dist-test-slave-23bp krb5kdc[24237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.188@KRBTEST.COM for kudu/127.1.182.189@KRBTEST.COM
Sep 28 02:10:09 dist-test-slave-23bp krb5kdc[24237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:10:09.706089 24470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.692088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:38497 (local address 127.0.0.1:48776)
0928 02:10:09.692341 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.692355 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:09.692444 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:09.692819 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:09.692826 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:09.693209 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:09.693342 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:09.693350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.694771 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.694777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:09.696045 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:09.696057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.696979 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.696983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:09.697210 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.697832 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:09.697861 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:09.699931 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:09.703723 (+ 3792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:09.703727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:09.703729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:09.704193 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:09.704624 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:09.704628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:09.704630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:09.704770 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:09.705348 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:09.705351 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:09.705466 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 02:10:09.705601 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 02:10:09.705802 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260928 02:10:09.706367 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.692087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48776 (local address 127.1.182.189:38497)
0928 02:10:09.692274 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:09.692279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:09.692435 (+ 156us) server_negotiation.cc:407] Connection header received
0928 02:10:09.692571 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:09.692578 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:09.692669 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:09.692765 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:09.693495 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.694640 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.696189 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.696840 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.696986 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.700137 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:09.700146 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:09.700152 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:09.700195 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:09.703578 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.704318 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.704323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.704326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.704502 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.704885 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.704891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.704895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.705224 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:09.705386 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:09.705730 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 02:10:09.705949 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:10:09.707442 24466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:10:09.708981 24321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.692097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.1.182.190:44847)
0928 02:10:09.692268 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:09.692274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:09.692530 (+ 256us) server_negotiation.cc:407] Connection header received
0928 02:10:09.692687 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:09.692690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:09.692771 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:09.692862 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:09.694390 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.695498 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.697161 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.697730 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.697871 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.702457 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:09.702462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:09.702465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:09.702497 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:09.705934 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.706872 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.706881 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.706886 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.707052 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.707429 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.707437 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.707440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.707777 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:09.708043 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:09.708461 (+ 418us) server_negotiation.cc:299] Negotiation successful
0928 02:10:09.708700 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 02:10:09 dist-test-slave-23bp krb5kdc[24237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.188@KRBTEST.COM for kudu/127.1.182.188@KRBTEST.COM
Sep 28 02:10:09 dist-test-slave-23bp krb5kdc[24237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.189@KRBTEST.COM
I20260928 02:10:09.720463 24469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.708456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.1.182.188:45333)
0928 02:10:09.708614 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:09.708623 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:09.708641 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:10:09.708687 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:09.708692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:09.708757 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:09.708864 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:09.709590 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.710364 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.711704 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.712110 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.712252 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.715234 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:09.715239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:09.715242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:09.715269 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:09.717843 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.718455 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.718463 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.718469 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.718623 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.718980 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.718988 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.718991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.719313 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:09.719458 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:09.719786 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 02:10:09.720125 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":167,"server-negotiator.queue_time_us":53}
I20260928 02:10:09.720470 24470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.708221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45333 (local address 127.0.0.1:46786)
0928 02:10:09.708399 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.708418 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:09.708511 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:09.708900 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:09.708904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:09.709321 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:09.709455 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:09.709463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.710494 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.710502 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:09.711571 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:09.711583 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.712250 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.712256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:09.712480 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.713060 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:09.713077 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:09.715007 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:09.717976 (+ 2969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:09.717983 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:09.717987 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:09.718342 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:09.718727 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:09.718734 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:09.718739 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:09.718869 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:09.719421 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:09.719425 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:09.719566 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 02:10:09.719699 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 02:10:09.719996 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":1}
I20260928 02:10:09.725049 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.710846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48786 (local address 127.1.182.189:38497)
0928 02:10:09.711025 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:09.711034 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:09.711053 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:10:09.711123 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:09.711128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:09.711194 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:09.711294 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:09.712529 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.713232 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.714388 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.714960 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.715113 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.719421 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:09.719426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:09.719429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:09.719460 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:09.722356 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.722994 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.723001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.723004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.723234 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.723615 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.723620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.723623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.723885 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:09.724159 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:09.724507 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 02:10:09.724738 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:10:09.725548 24466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958: Bootstrap starting.
I20260928 02:10:09.729391 24466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:09.730079 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:10:09.730643 24466 log.cc:824] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:09.733314 24466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958: No bootstrap required, opened a new log
I20260928 02:10:09.732674 24292 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43398
I20260928 02:10:09.740664 24466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44847 } } peers { permanent_uuid: "3a1d9f1f87d24dfb9085919adb9b820e" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } } peers { permanent_uuid: "0876a5c5647442f49ca9f06e28859958" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } }
I20260928 02:10:09.741012 24466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0876a5c5647442f49ca9f06e28859958, State: Initialized, Role: FOLLOWER
I20260928 02:10:09.741446 24466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44847 } } peers { permanent_uuid: "3a1d9f1f87d24dfb9085919adb9b820e" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } } peers { permanent_uuid: "0876a5c5647442f49ca9f06e28859958" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } }
I20260928 02:10:09.742429 24472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44847 } } peers { permanent_uuid: "3a1d9f1f87d24dfb9085919adb9b820e" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } } peers { permanent_uuid: "0876a5c5647442f49ca9f06e28859958" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } } }
I20260928 02:10:09.742815 24472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:10:09.743635 24466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:10:09.754205 24483 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:10:09.754359 24483 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:10:09.752430 24399 master_runner.cc:186] Error getting master registration for 127.1.182.190:44847: OK, instance_id { permanent_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" instance_seqno: 1790561408632688 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:10:09.763829 24399 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:10:09.774120 24317 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } attempt: 4
Sep 28 02:10:09 dist-test-slave-23bp krb5kdc[24237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561408, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for kudu/127.1.182.188@KRBTEST.COM
I20260928 02:10:09.790623 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.774908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45333 (local address 127.0.0.1:46796)
0928 02:10:09.775173 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.775195 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:09.775328 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:09.775738 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:09.775742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:09.776280 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:09.776500 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:09.776511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.777980 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.777985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:09.779468 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:09.779485 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.780310 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.780315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:09.780521 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.781164 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:09.781181 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:09.783473 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:09.787481 (+ 4008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:09.787486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:09.787488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:09.787856 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:09.788294 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:09.788301 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:09.788307 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:09.788558 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:09.789421 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:09.789436 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:09.789651 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 02:10:09.789835 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 02:10:09.790180 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 02:10:09.790691 24469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.774974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.1.182.188:45333)
0928 02:10:09.775185 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:09.775193 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:09.775293 (+ 100us) server_negotiation.cc:407] Connection header received
0928 02:10:09.775461 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:09.775467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:09.775551 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:09.775655 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:09.776664 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.777819 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.779604 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.780126 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.780290 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.783680 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:09.783687 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:09.783692 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:09.783731 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:09.787338 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.787989 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.787995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.787998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.788155 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.788706 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.788711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.788714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.789278 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:09.789460 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:09.789992 (+ 532us) server_negotiation.cc:299] Negotiation successful
0928 02:10:09.790297 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:10:09.796725 24317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9: Bootstrap starting.
I20260928 02:10:09.800707 24317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:09.802083 24317 log.cc:824] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:09.804894 24317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9: No bootstrap required, opened a new log
I20260928 02:10:09.812539 24317 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44847 } } peers { permanent_uuid: "3a1d9f1f87d24dfb9085919adb9b820e" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } } peers { permanent_uuid: "0876a5c5647442f49ca9f06e28859958" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } }
I20260928 02:10:09.812987 24317 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c55061bc4040fb88d7e3d3e3aed9e9, State: Initialized, Role: FOLLOWER
I20260928 02:10:09.813503 24317 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44847 } } peers { permanent_uuid: "3a1d9f1f87d24dfb9085919adb9b820e" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } } peers { permanent_uuid: "0876a5c5647442f49ca9f06e28859958" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } }
I20260928 02:10:09.814518 24486 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44847 } } peers { permanent_uuid: "3a1d9f1f87d24dfb9085919adb9b820e" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } } peers { permanent_uuid: "0876a5c5647442f49ca9f06e28859958" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } } }
I20260928 02:10:09.814963 24486 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:10:09.815838 24317 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:10:09.815964 24292 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 02:10:09.816193 24292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:09.819705 24292 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44847 } } peers { permanent_uuid: "3a1d9f1f87d24dfb9085919adb9b820e" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } } peers { permanent_uuid: "0876a5c5647442f49ca9f06e28859958" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } }
I20260928 02:10:09.821231 24292 leader_election.cc:288] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [CANDIDATE]: Term 1 election: Requested vote from peers 3a1d9f1f87d24dfb9085919adb9b820e (127.1.182.189:38497), 0876a5c5647442f49ca9f06e28859958 (127.1.182.188:45333)
I20260928 02:10:09.822391 24441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0876a5c5647442f49ca9f06e28859958"
I20260928 02:10:09.823029 24441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:09.827924 24441 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6c55061bc4040fb88d7e3d3e3aed9e9 in term 1.
I20260928 02:10:09.828881 24268 leader_election.cc:302] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0876a5c5647442f49ca9f06e28859958, d6c55061bc4040fb88d7e3d3e3aed9e9; no voters:
I20260928 02:10:09.829511 24486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:10:09.841048 24486 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [term 1 LEADER]: Becoming Leader. State: Replica: d6c55061bc4040fb88d7e3d3e3aed9e9, State: Running, Role: LEADER
I20260928 02:10:09.841993 24486 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44847 } } peers { permanent_uuid: "3a1d9f1f87d24dfb9085919adb9b820e" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } } peers { permanent_uuid: "0876a5c5647442f49ca9f06e28859958" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } }
I20260928 02:10:09.851298 24491 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6c55061bc4040fb88d7e3d3e3aed9e9. Latest consensus state: current_term: 1 leader_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44847 } } peers { permanent_uuid: "3a1d9f1f87d24dfb9085919adb9b820e" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } } peers { permanent_uuid: "0876a5c5647442f49ca9f06e28859958" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } } }
I20260928 02:10:09.851890 24491 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:09.858034 24500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:10:09.863055 24392 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } attempt: 5
W20260928 02:10:09.872035 24271 tablet.cc:2369] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:10:09.876446 24500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:10:09.886226 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.846367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48788 (local address 127.1.182.189:38497)
0928 02:10:09.846556 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:09.846562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:09.850870 (+ 4308us) server_negotiation.cc:407] Connection header received
0928 02:10:09.851914 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:09.851922 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:09.852027 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:09.852113 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:09.858834 (+ 6721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.859997 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.864493 (+ 4496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.865167 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.865283 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.877951 (+ 12668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:09.877960 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:09.877966 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:09.878014 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:09.881680 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.882991 (+ 1311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.882996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.883000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.883186 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.883895 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.883903 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.883906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.884340 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:09.884578 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:09.885554 (+ 976us) server_negotiation.cc:299] Negotiation successful
0928 02:10:09.885813 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 02:10:09.887601 24420 tablet.cc:2369] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:10:09.894137 24469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.855653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.1.182.188:45333)
0928 02:10:09.855826 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:09.855833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:09.863602 (+ 7769us) server_negotiation.cc:407] Connection header received
0928 02:10:09.863881 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:09.863886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:09.864009 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:09.864089 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:09.879387 (+ 15298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.880560 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.882484 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.883209 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.883321 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.886650 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:09.886658 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:09.886663 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:09.886702 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:09.890400 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.891192 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.891197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.891201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.891343 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.891784 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.891792 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.891798 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.892157 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:09.892427 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:09.893551 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0928 02:10:09.893797 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:10:09.895750 24510 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
Sep 28 02:10:09 dist-test-slave-23bp krb5kdc[24237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.189@KRBTEST.COM for kudu/127.1.182.188@KRBTEST.COM
I20260928 02:10:09.904429 24441 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958 [term 1 FOLLOWER]: Refusing update from remote peer d6c55061bc4040fb88d7e3d3e3aed9e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:10:09.905652 24486 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0876a5c5647442f49ca9f06e28859958" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:10:09.915693 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.863664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:45333 (local address 127.0.0.1:46814)
0928 02:10:09.863905 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:09.863924 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:09.864009 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:09.893314 (+ 29305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:09.893319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:09.893806 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:09.893999 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:09.894011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.898352 (+ 4341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.898359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:09.899819 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:09.899834 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.901455 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.901461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:09.901676 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.902302 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:09.902325 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:09.905132 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:09.912026 (+ 6894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:09.912032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:09.912047 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:09.912599 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:09.913284 (+ 685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:09.913289 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:09.913293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:09.913457 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:09.914592 (+ 1135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:09.914598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:09.914767 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 02:10:09.914937 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 02:10:09.915294 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260928 02:10:09.917827 24505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.864284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46814 (local address 127.1.182.188:45333)
0928 02:10:09.892740 (+ 28456us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:09.892745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:09.892795 (+ 50us) server_negotiation.cc:407] Connection header received
0928 02:10:09.892874 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:09.892880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:09.893011 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:09.894200 (+ 1189us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:09.894709 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.898146 (+ 3437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.900090 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.901173 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.902132 (+ 959us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.905489 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:09.905499 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:09.905506 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:09.905577 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:09.911783 (+ 6206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.912854 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.912859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.912864 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.913112 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.913645 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.913652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.913658 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.914386 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:09.916250 (+ 1864us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:09.916490 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 02:10:09.916873 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28247,"thread_start_us":27729,"threads_started":1}
I20260928 02:10:09.918296 24321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:09.848435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43414 (local address 127.1.182.190:44847)
0928 02:10:09.848598 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:09.848604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:09.856796 (+ 8192us) server_negotiation.cc:407] Connection header received
0928 02:10:09.857161 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:09.857167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:09.857249 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:09.857320 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:09.890390 (+ 33070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.891556 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.902292 (+ 10736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:09.903143 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:09.903260 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:09.906629 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:09.906637 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:09.906646 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:09.906682 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:09.913554 (+ 6872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.914538 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.914543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.914546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.914721 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:09.915220 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:09.915226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:09.915232 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:09.915669 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:09.917235 (+ 1566us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:09.917370 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 02:10:09.917630 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:10:09.899519 24250 master_runner.cc:186] Error getting master registration for 127.1.182.189:38497: OK, instance_id { permanent_uuid: "3a1d9f1f87d24dfb9085919adb9b820e" instance_seqno: 1790561409131390 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:10:09.926069 24250 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:10:09.929162 24472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6c55061bc4040fb88d7e3d3e3aed9e9. Latest consensus state: current_term: 1 leader_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44847 } } peers { permanent_uuid: "3a1d9f1f87d24dfb9085919adb9b820e" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } } peers { permanent_uuid: "0876a5c5647442f49ca9f06e28859958" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } } }
I20260928 02:10:09.929492 24472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:10:09.931010 24392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a1d9f1f87d24dfb9085919adb9b820e: Bootstrap starting.
I20260928 02:10:09.933598 24508 mvcc.cc:205] Tried to move back new op lower bound from 7334139534925996032 to 7334139534721880064. Current Snapshot: MvccSnapshot[applied={T|T < 7334139534925996032}]
I20260928 02:10:09.934079 24511 mvcc.cc:205] Tried to move back new op lower bound from 7334139534925996032 to 7334139534721880064. Current Snapshot: MvccSnapshot[applied={T|T < 7334139534925996032}]
I20260928 02:10:09.940567 24491 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44847 } } peers { permanent_uuid: "3a1d9f1f87d24dfb9085919adb9b820e" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } } peers { permanent_uuid: "0876a5c5647442f49ca9f06e28859958" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } } }
I20260928 02:10:09.940862 24491 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:09.942028 24500 catalog_manager.cc:1347] Generated new cluster ID: ecd47042a67b4a66b08df923c7917390
I20260928 02:10:09.942137 24500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:10:09.943387 24472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44847 } } peers { permanent_uuid: "3a1d9f1f87d24dfb9085919adb9b820e" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } } peers { permanent_uuid: "0876a5c5647442f49ca9f06e28859958" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } } }
I20260928 02:10:09.943640 24472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:10:09.953498 24392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a1d9f1f87d24dfb9085919adb9b820e: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:09.956255 24392 log.cc:824] T 00000000000000000000000000000000 P 3a1d9f1f87d24dfb9085919adb9b820e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:09.972569 24500 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:10:09.974496 24500 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:10:09.976102 24392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a1d9f1f87d24dfb9085919adb9b820e: No bootstrap required, opened a new log
I20260928 02:10:09.992743 24500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9: Generated new TSK 0
I20260928 02:10:09.994168 24500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:10:10.004441 24392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a1d9f1f87d24dfb9085919adb9b820e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44847 } } peers { permanent_uuid: "3a1d9f1f87d24dfb9085919adb9b820e" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } } peers { permanent_uuid: "0876a5c5647442f49ca9f06e28859958" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } }
I20260928 02:10:10.004904 24392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a1d9f1f87d24dfb9085919adb9b820e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a1d9f1f87d24dfb9085919adb9b820e, State: Initialized, Role: FOLLOWER
I20260928 02:10:10.005438 24392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a1d9f1f87d24dfb9085919adb9b820e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44847 } } peers { permanent_uuid: "3a1d9f1f87d24dfb9085919adb9b820e" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } } peers { permanent_uuid: "0876a5c5647442f49ca9f06e28859958" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } }
I20260928 02:10:10.015653 24516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a1d9f1f87d24dfb9085919adb9b820e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44847 } } peers { permanent_uuid: "3a1d9f1f87d24dfb9085919adb9b820e" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } } peers { permanent_uuid: "0876a5c5647442f49ca9f06e28859958" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } } }
I20260928 02:10:10.017134 24392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a1d9f1f87d24dfb9085919adb9b820e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:10:10.016776 24516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a1d9f1f87d24dfb9085919adb9b820e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:10:10.023680 24365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a1d9f1f87d24dfb9085919adb9b820e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:10.027379 24366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3a1d9f1f87d24dfb9085919adb9b820e"
I20260928 02:10:10.036446 24365 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a1d9f1f87d24dfb9085919adb9b820e [term 1 FOLLOWER]: Refusing update from remote peer d6c55061bc4040fb88d7e3d3e3aed9e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:10:10.041888 24366 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3a1d9f1f87d24dfb9085919adb9b820e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6c55061bc4040fb88d7e3d3e3aed9e9 in term 1.
I20260928 02:10:10.045574 24486 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a1d9f1f87d24dfb9085919adb9b820e" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:10:10.098176 24516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a1d9f1f87d24dfb9085919adb9b820e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6c55061bc4040fb88d7e3d3e3aed9e9. Latest consensus state: current_term: 1 leader_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44847 } } peers { permanent_uuid: "3a1d9f1f87d24dfb9085919adb9b820e" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } } peers { permanent_uuid: "0876a5c5647442f49ca9f06e28859958" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } } }
I20260928 02:10:10.098135 24525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a1d9f1f87d24dfb9085919adb9b820e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44847 } } peers { permanent_uuid: "3a1d9f1f87d24dfb9085919adb9b820e" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } } peers { permanent_uuid: "0876a5c5647442f49ca9f06e28859958" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } } }
I20260928 02:10:10.098559 24516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a1d9f1f87d24dfb9085919adb9b820e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:10:10.098629 24525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a1d9f1f87d24dfb9085919adb9b820e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:10:10.129477 24536 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260928 02:10:10.148978 24396 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:10.141892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.1.182.189:38497)
0928 02:10:10.142088 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:10.142094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:10.145951 (+ 3857us) server_negotiation.cc:407] Connection header received
0928 02:10:10.147404 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:10.147410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:10.147511 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:10.147584 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:10.148746 (+ 1162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48804: BlockingRecv error: recv got EOF from 127.0.0.1:48804 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 02:10:10.159569 24505 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:10.146729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46818 (local address 127.1.182.188:45333)
0928 02:10:10.146896 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:10.146901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:10.157767 (+ 10866us) server_negotiation.cc:407] Connection header received
0928 02:10:10.158016 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:10.158022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:10.158162 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:10.158284 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:10.159315 (+ 1031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46818: BlockingRecv error: recv got EOF from 127.0.0.1:46818 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 02:10:10.160409 24321 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:10.146047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43426 (local address 127.1.182.190:44847)
0928 02:10:10.146244 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:10.146251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:10.158571 (+ 12320us) server_negotiation.cc:407] Connection header received
0928 02:10:10.158840 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:10.158845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:10.158951 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:10.159039 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:10.160161 (+ 1122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43426: BlockingRecv error: recv got EOF from 127.0.0.1:43426 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 02:10:10.176712 24396 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:10.174725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48816 (local address 127.1.182.189:38497)
0928 02:10:10.174930 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:10.174936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:10.175044 (+ 108us) server_negotiation.cc:407] Connection header received
0928 02:10:10.175413 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:10.175418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:10.175511 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:10.175633 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:10.176420 (+ 787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48816: BlockingRecv error: recv got EOF from 127.0.0.1:48816 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 02:10:10.183817 24396 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:10.181751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48824 (local address 127.1.182.189:38497)
0928 02:10:10.181968 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:10.181976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:10.182477 (+ 501us) server_negotiation.cc:407] Connection header received
0928 02:10:10.182705 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:10.182711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:10.182796 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:10.182903 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:10.183651 (+ 748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48824: BlockingRecv error: recv got EOF from 127.0.0.1:48824 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 02:10:10.192576 24396 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:10.190497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48830 (local address 127.1.182.189:38497)
0928 02:10:10.190701 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:10.190711 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:10.191232 (+ 521us) server_negotiation.cc:407] Connection header received
0928 02:10:10.191440 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:10.191446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:10.191534 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:10.191651 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:10.192328 (+ 677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48830: BlockingRecv error: recv got EOF from 127.0.0.1:48830 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 02:10:10.201749 24396 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:10.199463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48844 (local address 127.1.182.189:38497)
0928 02:10:10.199726 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:10.199736 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:10.200242 (+ 506us) server_negotiation.cc:407] Connection header received
0928 02:10:10.200464 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:10.200468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:10.200535 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:10.200650 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:10.201520 (+ 870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48844: BlockingRecv error: recv got EOF from 127.0.0.1:48844 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
W20260928 02:10:10.219168 24396 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:10.216713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.1.182.189:38497)
0928 02:10:10.216933 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:10.216943 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:10.217592 (+ 649us) server_negotiation.cc:407] Connection header received
0928 02:10:10.217878 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:10.217884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:10.217957 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:10.218072 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:10.218894 (+ 822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48860: BlockingRecv error: recv got EOF from 127.0.0.1:48860 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 02:10:10.241096 24396 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:10.238839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48866 (local address 127.1.182.189:38497)
0928 02:10:10.239099 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:10.239109 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:10.239633 (+ 524us) server_negotiation.cc:407] Connection header received
0928 02:10:10.239901 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:10.239905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:10.239977 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:10.240103 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:10.240864 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48866: BlockingRecv error: recv got EOF from 127.0.0.1:48866 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 02:10:10.282459 24396 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:10.280160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48874 (local address 127.1.182.189:38497)
0928 02:10:10.280356 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:10.280363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:10.280910 (+ 547us) server_negotiation.cc:407] Connection header received
0928 02:10:10.281120 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:10.281126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:10.281219 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:10.281332 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:10.282182 (+ 850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48874: BlockingRecv error: recv got EOF from 127.0.0.1:48874 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 02:10:10.352537 24396 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:10.350456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.1.182.189:38497)
0928 02:10:10.350712 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:10.350719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:10.351134 (+ 415us) server_negotiation.cc:407] Connection header received
0928 02:10:10.351333 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:10.351339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:10.351442 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:10.351591 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:10.352303 (+ 712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48890: BlockingRecv error: recv got EOF from 127.0.0.1:48890 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260928 02:10:10.486032 24396 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:10.483911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48896 (local address 127.1.182.189:38497)
0928 02:10:10.484138 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:10.484145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:10.484513 (+ 368us) server_negotiation.cc:407] Connection header received
0928 02:10:10.484775 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:10.484780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:10.484850 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:10.484937 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:10.485764 (+ 827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48896: BlockingRecv error: recv got EOF from 127.0.0.1:48896 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:10:10.567670 24486 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c55061bc4040fb88d7e3d3e3aed9e9" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 44847 } } peers { permanent_uuid: "3a1d9f1f87d24dfb9085919adb9b820e" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 38497 } } peers { permanent_uuid: "0876a5c5647442f49ca9f06e28859958" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 45333 } } }
I20260928 02:10:10.568078 24486 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c55061bc4040fb88d7e3d3e3aed9e9 [sys.catalog]: This master's current role is: LEADER
W20260928 02:10:10.748555 24396 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:10.746311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48902 (local address 127.1.182.189:38497)
0928 02:10:10.746545 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:10.746552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:10.746900 (+ 348us) server_negotiation.cc:407] Connection header received
0928 02:10:10.747188 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:10.747198 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:10.747307 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:10.747424 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:10.748310 (+ 886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48902: BlockingRecv error: recv got EOF from 127.0.0.1:48902 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 02:10:10.756909 24483 catalog_manager.cc:1259] Loaded cluster ID: ecd47042a67b4a66b08df923c7917390
I20260928 02:10:10.757022 24483 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958: loading cluster ID for follower catalog manager: success
I20260928 02:10:10.760751 24483 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958: acquiring CA information for follower catalog manager: success
I20260928 02:10:10.763161 24483 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0876a5c5647442f49ca9f06e28859958: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:10:11.134513 24536 catalog_manager.cc:1259] Loaded cluster ID: ecd47042a67b4a66b08df923c7917390
I20260928 02:10:11.134663 24536 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3a1d9f1f87d24dfb9085919adb9b820e: loading cluster ID for follower catalog manager: success
I20260928 02:10:11.138908 24536 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3a1d9f1f87d24dfb9085919adb9b820e: acquiring CA information for follower catalog manager: success
I20260928 02:10:11.141503 24536 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3a1d9f1f87d24dfb9085919adb9b820e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:10:11.280393 24601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:11.272985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48910 (local address 127.1.182.189:38497)
0928 02:10:11.274300 (+ 1315us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:11.274308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:11.274584 (+ 276us) server_negotiation.cc:407] Connection header received
0928 02:10:11.274833 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:11.274839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:11.275023 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:11.275221 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:11.275673 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:11.276506 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:11.277860 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:11.278365 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:11.278526 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:11.279161 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:11.279754 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:11.279929 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:11.279984 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 02:10:11.280078 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1117,"thread_start_us":978,"threads_started":1}
I20260928 02:10:11.292541 24608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:11.286754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46826 (local address 127.1.182.188:45333)
0928 02:10:11.287366 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:11.287373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:11.287392 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:10:11.287525 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:11.287528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:11.287687 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:11.287861 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:11.288268 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:11.289001 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:11.290306 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:11.290717 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:11.290851 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:11.291396 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:11.291909 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:11.292083 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:11.292163 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 02:10:11.292245 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":386,"threads_started":1}
I20260928 02:10:11.318931 24601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:11.299547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48926 (local address 127.1.182.189:38497)
0928 02:10:11.300034 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:11.300051 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:11.311582 (+ 11531us) server_negotiation.cc:407] Connection header received
0928 02:10:11.311858 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:11.311869 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:11.311962 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:11.312024 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:11.312811 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:11.314009 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:11.316575 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:11.317272 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:11.317369 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:11.318075 (+ 706us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:11.318210 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:11.318275 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:11.318515 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 02:10:11.318628 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357}
I20260928 02:10:11.321198 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:11.299734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43442 (local address 127.1.182.190:44847)
0928 02:10:11.310383 (+ 10649us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:11.310390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:11.310827 (+ 437us) server_negotiation.cc:407] Connection header received
0928 02:10:11.311160 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:11.311169 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:11.311422 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:11.312229 (+ 807us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:11.312438 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:11.313947 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:11.317359 (+ 3412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:11.318351 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:11.319181 (+ 830us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:11.319438 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:11.320022 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:11.320680 (+ 658us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:11.320734 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 02:10:11.320874 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10472,"thread_start_us":10327,"threads_started":1}
I20260928 02:10:11.326141 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 24250
I20260928 02:10:11.326432 24608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:11.311856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46838 (local address 127.1.182.188:45333)
0928 02:10:11.312062 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:11.312068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:11.319927 (+ 7859us) server_negotiation.cc:407] Connection header received
0928 02:10:11.320115 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:11.320123 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:11.320236 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:11.320326 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:11.320668 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:11.321658 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:11.324435 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:11.324988 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:11.325131 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:11.325699 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:11.325827 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:11.325918 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:11.326095 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 02:10:11.326173 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:10:11.327214 24312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:10:11.446137 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 24250
I20260928 02:10:11.468251 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 24324
I20260928 02:10:11.469424 24387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:10:11.593117 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 24324
I20260928 02:10:11.613054 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 24399
I20260928 02:10:11.614261 24461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:10:11.722791 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 24399
2026-09-28T02:10:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4926 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:10:11 dist-test-slave-23bp krb5kdc[24622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:10:11 dist-test-slave-23bp krb5kdc[24622](info): set up 2 sockets
Sep 28 02:10:11 dist-test-slave-23bp krb5kdc[24622](info): commencing operation
krb5kdc: starting...
W20260928 02:10:12.846020 1754 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/test-user.keytab.
Sep 28 02:10:13 dist-test-slave-23bp krb5kdc[24622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:10:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:10:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.182.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
Entry for principal kudu/127.1.182.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab.
I20260928 02:10:13.111723 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:38039
--webserver_interface=127.1.182.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:34463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:38039,127.1.182.189:39537,127.1.182.188:33211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:13.444621 24636 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:10:13.444989 24636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:13.445106 24636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:13.445240 24636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:13.452736 24636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:10:13.452829 24636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:13.452929 24636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:13.453022 24636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:10:13.453130 24636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:10:13.463156 24636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:34463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:38039,127.1.182.189:39537,127.1.182.188:33211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.190.keytab
--principal=kudu/127.1.182.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.190:38039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.182.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:13.464551 24636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:13.465911 24636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:13.478495 24641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:13.478555 24642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:13.478472 24644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:13.479375 24636 server_base.cc:1034] running on GCE node
I20260928 02:10:13.480136 24636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:13.481480 24636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:13.482736 24636 hybrid_clock.cc:648] HybridClock initialized: now 1790561413482678 us; error 52 us; skew 500 ppm
I20260928 02:10:13.483119 24636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:13 dist-test-slave-23bp krb5kdc[24622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561413, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:13.486249 24636 init.cc:377] Logged in from keytab as kudu/127.1.182.190@KRBTEST.COM (short username kudu)
I20260928 02:10:13.487782 24636 webserver.cc:460] Webserver started at http://127.1.182.190:42619/ using document root <none> and password file <none>
I20260928 02:10:13.488720 24636 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:13.488828 24636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:13.489190 24636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:13.492259 24636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-0/data/instance:
uuid: "974ab0ceedc546bdaee6611a93e28501"
format_stamp: "Formatted at 2026-09-28 02:10:13 on dist-test-slave-23bp"
server_key: "432086cb887e6d669b7daf4a4170779a"
server_key_iv: "855fb46c86b2ccff1115d43b73397e2c"
server_key_version: "encryptionkey@0"
I20260928 02:10:13.493074 24636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-0/wal/instance:
uuid: "974ab0ceedc546bdaee6611a93e28501"
format_stamp: "Formatted at 2026-09-28 02:10:13 on dist-test-slave-23bp"
server_key: "432086cb887e6d669b7daf4a4170779a"
server_key_iv: "855fb46c86b2ccff1115d43b73397e2c"
server_key_version: "encryptionkey@0"
I20260928 02:10:13.498267 24636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260928 02:10:13.501675 24651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:13.503050 24636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:10:13.503301 24636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-0/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-0/wal
uuid: "974ab0ceedc546bdaee6611a93e28501"
format_stamp: "Formatted at 2026-09-28 02:10:13 on dist-test-slave-23bp"
server_key: "432086cb887e6d669b7daf4a4170779a"
server_key_iv: "855fb46c86b2ccff1115d43b73397e2c"
server_key_version: "encryptionkey@0"
I20260928 02:10:13.503520 24636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:13.520823 24636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:13.525066 24636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:13.525352 24636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:13.544291 24636 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.190:38039
I20260928 02:10:13.544313 24702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.190:38039 every 8 connection(s)
I20260928 02:10:13.545691 24636 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-0/data/info.pb
I20260928 02:10:13.547078 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 24636
I20260928 02:10:13.547264 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-0/wal/instance
I20260928 02:10:13.547835 1754 external_mini_cluster.cc:1428] Setting key 690aace1a254474cb15785606b5a5db0
I20260928 02:10:13.550442 24703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:13.560276 24703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38039 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:10:13 dist-test-slave-23bp krb5kdc[24622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561413, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.182.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.189@KRBTEST.COM" created.
I20260928 02:10:13.584528 24707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:13.563635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60396 (local address 127.1.182.190:38039)
0928 02:10:13.565304 (+ 1669us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:13.565313 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:13.565364 (+ 51us) server_negotiation.cc:407] Connection header received
0928 02:10:13.567162 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:13.567173 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:13.567502 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:13.567713 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:13.568944 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:13.570879 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:13.572414 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:13.573416 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:13.573623 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:13.577050 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:13.577064 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:13.577077 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:13.577106 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:13.580047 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:13.580753 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:13.580760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:13.580762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:13.580904 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:13.581316 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:13.581327 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:13.581330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:13.581822 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:13.582037 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:13.582574 (+ 537us) server_negotiation.cc:299] Negotiation successful
0928 02:10:13.582993 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":1460,"thread_start_us":1244,"threads_started":1}
I20260928 02:10:13.584569 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:13.563640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:38039 (local address 127.0.0.1:60396)
0928 02:10:13.564680 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:13.564743 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:13.566714 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:13.567758 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:13.567764 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:13.568365 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:13.568716 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:13.568734 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:13.571056 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:13.571060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:13.572257 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:13.572270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:13.573689 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:13.573694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:13.573818 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:13.574503 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:13.574534 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:13.576823 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:13.580191 (+ 3368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:13.580204 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:13.580217 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:13.580604 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:13.581025 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:13.581032 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:13.581038 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:13.581201 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:13.581962 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:13.581970 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:13.582192 (+ 222us) client_negotiation.cc:769] Sending connection context
0928 02:10:13.582441 (+ 249us) client_negotiation.cc:240] Negotiation successful
0928 02:10:13.583132 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":146,"thread_start_us":367,"threads_started":1}
I20260928 02:10:13.587597 24703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:10:13.588605 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:13.588232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39537 (local address 127.0.0.1:56750)
0928 02:10:13.588384 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:13.588487 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:39537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 02:10:13.589010 24654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.182.189:39537: connect: Connection refused (error 111)
I20260928 02:10:13.590476 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:13.590151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39537 (local address 127.0.0.1:56766)
0928 02:10:13.590300 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:13.590362 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:39537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 02:10:13.590759 24703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.189:39537: Network error: Client connection negotiation failed: client connection to 127.1.182.189:39537: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.189.keytab.
Entry for principal kudu/127.1.182.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.189.keytab.
I20260928 02:10:13.611855 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.189:39537
--webserver_interface=127.1.182.189
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.189.keytab
--principal=kudu/127.1.182.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:34463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:38039,127.1.182.189:39537,127.1.182.188:33211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:10:13.640101 24703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } attempt: 1
I20260928 02:10:13.641341 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:13.640906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39537 (local address 127.0.0.1:56772)
0928 02:10:13.641101 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:13.641207 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:39537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 02:10:13.642288 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:13.641948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39537 (local address 127.0.0.1:56780)
0928 02:10:13.642089 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:13.642167 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:39537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 02:10:13.642459 24703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.189:39537: Network error: Client connection negotiation failed: client connection to 127.1.182.189:39537: connect: Connection refused (error 111)
I20260928 02:10:13.710716 24703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } attempt: 2
I20260928 02:10:13.712026 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:13.711519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39537 (local address 127.0.0.1:56786)
0928 02:10:13.711778 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:13.711872 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:39537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260928 02:10:13.713078 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:13.712737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39537 (local address 127.0.0.1:56802)
0928 02:10:13.712866 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:13.712948 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:39537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 02:10:13.713347 24703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.189:39537: Network error: Client connection negotiation failed: client connection to 127.1.182.189:39537: connect: Connection refused (error 111)
I20260928 02:10:13.804629 24703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } attempt: 3
I20260928 02:10:13.805897 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:13.805447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39537 (local address 127.0.0.1:56808)
0928 02:10:13.805665 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:13.805761 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:39537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 02:10:13.807225 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:13.806763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39537 (local address 127.0.0.1:56818)
0928 02:10:13.806891 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:13.806999 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:39537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 02:10:13.807523 24703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.189:39537: Network error: Client connection negotiation failed: client connection to 127.1.182.189:39537: connect: Connection refused (error 111)
I20260928 02:10:13.950773 24703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } attempt: 4
I20260928 02:10:13.952105 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:13.951587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39537 (local address 127.0.0.1:56820)
0928 02:10:13.951877 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:13.951968 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:39537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260928 02:10:13.952811 24710 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260928 02:10:13.953262 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:13.952904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39537 (local address 127.0.0.1:56830)
0928 02:10:13.953038 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:13.953120 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.189:39537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 02:10:13.953364 24710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:13.953482 24710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:13.953446 24703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.189:39537: Network error: Client connection negotiation failed: client connection to 127.1.182.189:39537: connect: Connection refused (error 111)
W20260928 02:10:13.953617 24710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:13.961133 24710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:10:13.961223 24710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:13.961280 24710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:13.961330 24710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:10:13.961385 24710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:10:13.971417 24710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:34463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:38039,127.1.182.189:39537,127.1.182.188:33211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.189.keytab
--principal=kudu/127.1.182.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.189:39537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.182.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:13.972671 24710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:13.973953 24710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:13.985972 24718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:13.986114 24715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:13.985939 24716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:13.987190 24710 server_base.cc:1034] running on GCE node
I20260928 02:10:13.987937 24710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:13.989418 24710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:13.990639 24710 hybrid_clock.cc:648] HybridClock initialized: now 1790561413990554 us; error 56 us; skew 500 ppm
I20260928 02:10:13.990978 24710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:13 dist-test-slave-23bp krb5kdc[24622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561413, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:13.994161 24710 init.cc:377] Logged in from keytab as kudu/127.1.182.189@KRBTEST.COM (short username kudu)
I20260928 02:10:13.995823 24710 webserver.cc:460] Webserver started at http://127.1.182.189:35433/ using document root <none> and password file <none>
I20260928 02:10:13.996618 24710 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:13.996752 24710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:13.997076 24710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:14.000306 24710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-1/data/instance:
uuid: "0e70313e1ff846be9a0ffafd316f607c"
format_stamp: "Formatted at 2026-09-28 02:10:13 on dist-test-slave-23bp"
server_key: "3b2507074a8ccfb9e3155b75e1ff0080"
server_key_iv: "e6f7d3d18deff87d2a9057a2463cf319"
server_key_version: "encryptionkey@0"
I20260928 02:10:14.001145 24710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-1/wal/instance:
uuid: "0e70313e1ff846be9a0ffafd316f607c"
format_stamp: "Formatted at 2026-09-28 02:10:13 on dist-test-slave-23bp"
server_key: "3b2507074a8ccfb9e3155b75e1ff0080"
server_key_iv: "e6f7d3d18deff87d2a9057a2463cf319"
server_key_version: "encryptionkey@0"
I20260928 02:10:14.006637 24710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:10:14.010195 24725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:14.011659 24710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:10:14.011847 24710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-1/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-1/wal
uuid: "0e70313e1ff846be9a0ffafd316f607c"
format_stamp: "Formatted at 2026-09-28 02:10:13 on dist-test-slave-23bp"
server_key: "3b2507074a8ccfb9e3155b75e1ff0080"
server_key_iv: "e6f7d3d18deff87d2a9057a2463cf319"
server_key_version: "encryptionkey@0"
I20260928 02:10:14.012070 24710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:14.035962 24710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:14.040339 24710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:14.040657 24710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:14.059203 24710 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.189:39537
I20260928 02:10:14.059221 24776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.189:39537 every 8 connection(s)
I20260928 02:10:14.060592 24710 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-1/data/info.pb
I20260928 02:10:14.064329 24777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:14.067781 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 24710
I20260928 02:10:14.067965 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-1/wal/instance
I20260928 02:10:14.068429 1754 external_mini_cluster.cc:1428] Setting key 110f2d2d60a6e593c93f715fcbd52aaa
I20260928 02:10:14.071430 24777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38039 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:10:14 dist-test-slave-23bp krb5kdc[24622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561413, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.189@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.182.188@KRBTEST.COM; defaulting to no policy
I20260928 02:10:14.094261 24707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.074317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60400 (local address 127.1.182.190:38039)
0928 02:10:14.074570 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:14.074580 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:14.075803 (+ 1223us) server_negotiation.cc:407] Connection header received
0928 02:10:14.077728 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:14.077738 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:14.077834 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:14.077961 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:14.079581 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.080688 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.082233 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.082969 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.083145 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.086940 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:14.086949 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:14.086958 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:14.087014 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:14.090957 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.091852 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.091860 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.091865 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.092034 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.092460 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.092465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.092468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.092873 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:14.093059 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:14.093849 (+ 790us) server_negotiation.cc:299] Negotiation successful
0928 02:10:14.094031 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:10:14.095409 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.074764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:38039 (local address 127.0.0.1:60400)
0928 02:10:14.075620 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.075666 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:14.077430 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:14.078163 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:14.078171 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:14.078752 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:14.079397 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:14.079415 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.080855 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.080859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:14.082068 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:14.082080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.083247 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.083252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:14.083477 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.084234 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:14.084257 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:14.086688 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:14.091184 (+ 4496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:14.091191 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:14.091206 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:14.091694 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:14.092151 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:14.092154 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:14.092156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:14.092330 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:14.093012 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:14.093017 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:14.093374 (+ 357us) client_negotiation.cc:769] Sending connection context
0928 02:10:14.093691 (+ 317us) client_negotiation.cc:240] Negotiation successful
0928 02:10:14.094089 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":374,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.182.188@KRBTEST.COM" created.
I20260928 02:10:14.097301 24777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:10:14 dist-test-slave-23bp krb5kdc[24622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561413, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.189@KRBTEST.COM for kudu/127.1.182.189@KRBTEST.COM
I20260928 02:10:14.112634 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.097995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39537 (local address 127.0.0.1:56844)
0928 02:10:14.098220 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.098234 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:14.098318 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:14.099414 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:14.099418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:14.099875 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:14.100037 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:14.100045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.101391 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.101398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:14.102238 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:14.102249 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.103062 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.103086 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:14.103273 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.103907 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:14.103923 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:14.105915 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:14.110192 (+ 4277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:14.110196 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:14.110199 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:14.110549 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:14.110959 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:14.110962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:14.110964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:14.111196 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:14.111938 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:14.111941 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:14.112076 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 02:10:14.112186 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:10:14.112378 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260928 02:10:14.113142 24782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.098331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56844 (local address 127.1.182.189:39537)
0928 02:10:14.098954 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:14.098960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:14.098984 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:10:14.099026 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:14.099031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:14.099271 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:14.099428 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:14.100208 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.101258 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.102371 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.102914 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.103052 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.106124 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:14.106135 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:14.106148 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:14.106181 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:14.110049 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.110682 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.110693 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.110699 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.110853 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.111312 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.111319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.111325 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.111821 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:14.112032 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:14.112350 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 02:10:14.112641 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":348,"threads_started":1}
I20260928 02:10:14.115598 24777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:10:14.116843 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.116341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:33211 (local address 127.0.0.1:57180)
0928 02:10:14.116519 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.116698 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 02:10:14.117116 24726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
I20260928 02:10:14.118672 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.118359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:33211 (local address 127.0.0.1:57188)
0928 02:10:14.118510 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.118580 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:10:14.118901 24777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:33211: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.182.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.188.keytab.
Entry for principal kudu/127.1.182.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.188.keytab.
I20260928 02:10:14.130638 1754 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
/tmp/dist-test-tasktsFKfV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.188:33211
--webserver_interface=127.1.182.188
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.188.keytab
--principal=kudu/127.1.182.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.182.148:34463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.182.190:38039,127.1.182.189:39537,127.1.182.188:33211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:10:14.168197 24777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } attempt: 1
I20260928 02:10:14.169463 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.168954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:33211 (local address 127.0.0.1:57196)
0928 02:10:14.169186 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.169306 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260928 02:10:14.170521 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.170188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:33211 (local address 127.0.0.1:57202)
0928 02:10:14.170335 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.170410 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:10:14.170781 24777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:33211: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
I20260928 02:10:14.239022 24777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } attempt: 2
I20260928 02:10:14.240377 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.239917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:33211 (local address 127.0.0.1:57206)
0928 02:10:14.240131 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.240226 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 02:10:14.241456 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.241127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:33211 (local address 127.0.0.1:57210)
0928 02:10:14.241269 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.241348 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:10:14.241727 24777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:33211: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
I20260928 02:10:14.252693 24703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } attempt: 5
Sep 28 02:10:14 dist-test-slave-23bp krb5kdc[24622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561413, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for kudu/127.1.182.189@KRBTEST.COM
I20260928 02:10:14.269439 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.253465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39537 (local address 127.0.0.1:56850)
0928 02:10:14.253668 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.253690 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:14.253830 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:14.254300 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:14.254306 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:14.254919 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:14.255231 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:14.255248 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.256709 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.256717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:14.258356 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:14.258376 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.259493 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.259502 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:14.259812 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.260559 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:14.260587 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:14.263365 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:14.266933 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:14.266937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:14.266940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:14.267507 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:14.267906 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:14.267909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:14.267911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:14.268042 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:14.268687 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:14.268693 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:14.268847 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 02:10:14.268964 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 02:10:14.269197 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 02:10:14.269644 24782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.253599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56850 (local address 127.1.182.189:39537)
0928 02:10:14.253837 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:14.253844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:14.253872 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:10:14.253990 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:14.253998 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:14.254123 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:14.254249 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:14.255412 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.256490 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.258555 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.259311 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.259492 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.263565 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:14.263571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:14.263575 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:14.263615 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:14.266781 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.267638 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.267644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.267647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.267794 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.268156 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.268160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.268164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.268536 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:14.268686 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:14.269115 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 02:10:14.269339 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 02:10:14.270774 24703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:10:14.271893 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.271526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:33211 (local address 127.0.0.1:57218)
0928 02:10:14.271694 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.271770 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 02:10:14.272866 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.272547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:33211 (local address 127.0.0.1:57226)
0928 02:10:14.272703 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.272778 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 02:10:14.273124 24703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:33211: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
I20260928 02:10:14.324357 24703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } attempt: 1
I20260928 02:10:14.325678 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.325033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:33211 (local address 127.0.0.1:57230)
0928 02:10:14.325317 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.325443 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 02:10:14.326747 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.326345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:33211 (local address 127.0.0.1:57236)
0928 02:10:14.326523 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.326611 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 02:10:14.326977 24703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:33211: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
I20260928 02:10:14.332960 24777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } attempt: 3
I20260928 02:10:14.333953 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.333525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:33211 (local address 127.0.0.1:57248)
0928 02:10:14.333736 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.333813 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 02:10:14.334977 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.334579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:33211 (local address 127.0.0.1:57258)
0928 02:10:14.334799 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.334866 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260928 02:10:14.335247 24777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:33211: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
I20260928 02:10:14.395272 24703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } attempt: 2
I20260928 02:10:14.396538 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.395950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:33211 (local address 127.0.0.1:57272)
0928 02:10:14.396203 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.396315 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 02:10:14.397668 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.397285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:33211 (local address 127.0.0.1:57276)
0928 02:10:14.397443 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.397540 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 02:10:14.397962 24703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:33211: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
I20260928 02:10:14.478479 24777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } attempt: 4
I20260928 02:10:14.479689 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.479198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:33211 (local address 127.0.0.1:57280)
0928 02:10:14.479438 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.479534 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 02:10:14.480693 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.480371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:33211 (local address 127.0.0.1:57288)
0928 02:10:14.480524 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.480596 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 02:10:14.480945 24777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:33211: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
W20260928 02:10:14.485409 24785 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:10:14.485749 24785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:14.485845 24785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:14.486027 24785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:14.493472 24785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:10:14.493557 24785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:14.493655 24785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:14.493739 24785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:10:14.493853 24785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:10:14.503962 24785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.182.148:34463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.182.190:38039,127.1.182.189:39537,127.1.182.188:33211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/krb5kdc/kudu_127.1.182.188.keytab
--principal=kudu/127.1.182.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.182.188:33211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.182.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:14.504194 24703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } attempt: 3
I20260928 02:10:14.505374 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.504824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:33211 (local address 127.0.0.1:57300)
0928 02:10:14.505073 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.505182 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 02:10:14.505618 24785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:14.506484 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.506070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:33211 (local address 127.0.0.1:57314)
0928 02:10:14.506276 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.506357 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260928 02:10:14.506729 24703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.182.188:33211: Network error: Client connection negotiation failed: client connection to 127.1.182.188:33211: connect: Connection refused (error 111)
I20260928 02:10:14.507284 24785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:14.519788 24790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:14.519765 24791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:14.519846 24793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:14.520637 24785 server_base.cc:1034] running on GCE node
I20260928 02:10:14.521519 24785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:14.522922 24785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:14.524196 24785 hybrid_clock.cc:648] HybridClock initialized: now 1790561414524093 us; error 62 us; skew 500 ppm
I20260928 02:10:14.524549 24785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:14 dist-test-slave-23bp krb5kdc[24622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561414, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:14.527755 24785 init.cc:377] Logged in from keytab as kudu/127.1.182.188@KRBTEST.COM (short username kudu)
I20260928 02:10:14.529371 24785 webserver.cc:460] Webserver started at http://127.1.182.188:45057/ using document root <none> and password file <none>
I20260928 02:10:14.530117 24785 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:14.530246 24785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:14.530571 24785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:14.533598 24785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-2/data/instance:
uuid: "5dfe9438f3a24a04b92778386853bb4d"
format_stamp: "Formatted at 2026-09-28 02:10:14 on dist-test-slave-23bp"
server_key: "4fb1c052027f344dbb6e9229717cb845"
server_key_iv: "a521364ee3dd4f98f1fe20d3c4c21326"
server_key_version: "encryptionkey@0"
I20260928 02:10:14.534371 24785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-2/wal/instance:
uuid: "5dfe9438f3a24a04b92778386853bb4d"
format_stamp: "Formatted at 2026-09-28 02:10:14 on dist-test-slave-23bp"
server_key: "4fb1c052027f344dbb6e9229717cb845"
server_key_iv: "a521364ee3dd4f98f1fe20d3c4c21326"
server_key_version: "encryptionkey@0"
I20260928 02:10:14.539551 24785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:10:14.542989 24800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:14.544488 24785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:10:14.544670 24785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-2/data,/tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-2/wal
uuid: "5dfe9438f3a24a04b92778386853bb4d"
format_stamp: "Formatted at 2026-09-28 02:10:14 on dist-test-slave-23bp"
server_key: "4fb1c052027f344dbb6e9229717cb845"
server_key_iv: "a521364ee3dd4f98f1fe20d3c4c21326"
server_key_version: "encryptionkey@0"
I20260928 02:10:14.544884 24785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:14.560304 24785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:14.564311 24785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:14.564594 24785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:14.583519 24785 rpc_server.cc:307] RPC server started. Bound to: 127.1.182.188:33211
I20260928 02:10:14.583554 24852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.182.188:33211 every 8 connection(s)
I20260928 02:10:14.584910 24785 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-2/data/info.pb
I20260928 02:10:14.587263 1754 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu as pid 24785
I20260928 02:10:14.587450 1754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktsFKfV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561052730740-1754-0/minicluster-data/master-2/wal/instance
I20260928 02:10:14.587908 1754 external_mini_cluster.cc:1428] Setting key 659bea7828551e679144b8035b56926f
I20260928 02:10:14.589052 24853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:14.598409 24853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38039 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:10:14 dist-test-slave-23bp krb5kdc[24622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.188@KRBTEST.COM
Sep 28 02:10:14 dist-test-slave-23bp krb5kdc[24622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561414, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.188@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
I20260928 02:10:14.611742 24856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.589685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.1.182.188:33211)
0928 02:10:14.590455 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:14.590467 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:14.590508 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:10:14.592132 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:14.592154 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:14.592573 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:14.592908 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:14.594714 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.596214 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.597967 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.598634 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.598767 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.603804 (+ 5037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:14.603812 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:14.603822 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:14.603860 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:14.607256 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.607960 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.607972 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.607984 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.608209 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.608660 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.608666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.608670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.609283 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:14.609540 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:14.609972 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 02:10:14.610351 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":374,"threads_started":1}
I20260928 02:10:14.618294 24858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.601653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60412 (local address 127.1.182.190:38039)
0928 02:10:14.602994 (+ 1341us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:14.602999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:14.603015 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:10:14.603105 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:14.603111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:14.603181 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:14.603298 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:14.604454 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.605742 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.607389 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.607903 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.608065 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.611826 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:14.611830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:14.611834 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:14.611890 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:14.615218 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.616065 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.616071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.616075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.616214 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.616661 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.616667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.616670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.617038 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:14.617205 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:14.617649 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 02:10:14.617858 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1218,"thread_start_us":1062,"threads_started":1}
I20260928 02:10:14.618292 24857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.601583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.190:38039 (local address 127.0.0.1:60412)
0928 02:10:14.602322 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.602367 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:14.602530 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:14.603356 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:14.603363 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:14.604001 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:14.604275 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:14.604286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.605911 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.605915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:14.607235 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:14.607247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.608147 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.608153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:14.608306 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.608986 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:14.609008 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:14.611621 (+ 2613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:14.615387 (+ 3766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:14.615396 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:14.615400 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:14.615930 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:14.616331 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:14.616336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:14.616340 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:14.616549 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:14.617181 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:14.617188 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:14.617371 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 02:10:14.617550 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 02:10:14.617853 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":375,"threads_started":1}
I20260928 02:10:14.620153 24853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:10:14 dist-test-slave-23bp krb5kdc[24622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.190@KRBTEST.COM
Sep 28 02:10:14 dist-test-slave-23bp krb5kdc[24622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561414, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.188@KRBTEST.COM for kudu/127.1.182.189@KRBTEST.COM
I20260928 02:10:14.631826 24859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.614852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60422 (local address 127.1.182.190:38039)
0928 02:10:14.615510 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:14.615514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:14.615531 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:10:14.615567 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:14.615570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:14.615621 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:14.615757 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:14.617054 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.617888 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.619579 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.620081 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.620213 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.625407 (+ 5194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:14.625414 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:14.625418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:14.625452 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:14.628734 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.629427 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.629431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.629433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.629620 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.629985 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.629990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.629994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.630388 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:14.630646 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:14.631142 (+ 496us) server_negotiation.cc:299] Negotiation successful
0928 02:10:14.631388 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":342,"threads_started":1}
I20260928 02:10:14.635119 24857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.620894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.189:39537 (local address 127.0.0.1:56858)
0928 02:10:14.621074 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.621094 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:14.621188 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:14.621557 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:14.621560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:14.621872 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:14.622012 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:14.622019 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.623502 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.623505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:14.624713 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:14.624726 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.625603 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.625610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:14.625806 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.626382 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:14.626399 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:14.628654 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:14.632465 (+ 3811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:14.632478 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:14.632482 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:14.632845 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:14.633237 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:14.633242 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:14.633246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:14.633431 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:14.634116 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:14.634120 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:14.634276 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 02:10:14.634432 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 02:10:14.634719 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 02:10:14.635178 24782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.621012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56858 (local address 127.1.182.189:39537)
0928 02:10:14.621209 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:14.621214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:14.621252 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:10:14.621319 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:14.621323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:14.621412 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:14.621516 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:14.622164 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.623353 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.624861 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.625451 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.625588 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.628850 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:14.628855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:14.628858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:14.628891 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:14.632287 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.632972 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.632977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.632980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.633122 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.633570 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.633575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.633578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.633949 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:14.634155 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:14.634532 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 02:10:14.634747 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:10:14.636153 24853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:10:14 dist-test-slave-23bp krb5kdc[24622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.182.189@KRBTEST.COM
Sep 28 02:10:14 dist-test-slave-23bp krb5kdc[24622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561414, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.188@KRBTEST.COM for kudu/127.1.182.188@KRBTEST.COM
I20260928 02:10:14.650359 24856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.637001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57330 (local address 127.1.182.188:33211)
0928 02:10:14.637153 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:14.637157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:14.637317 (+ 160us) server_negotiation.cc:407] Connection header received
0928 02:10:14.637467 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:14.637472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:14.637540 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:14.637641 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:14.638341 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.639211 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.640510 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.641219 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.641354 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.644605 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:14.644610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:14.644613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:14.644640 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:14.647421 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.648045 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.648049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.648053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.648235 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.648615 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.648620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.648623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.648944 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:14.649054 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:14.649611 (+ 557us) server_negotiation.cc:299] Negotiation successful
0928 02:10:14.649889 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":54}
I20260928 02:10:14.650383 24857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.636962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:33211 (local address 127.0.0.1:57330)
0928 02:10:14.637234 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.637254 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:14.637349 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:14.637708 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:14.637717 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:14.638063 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:14.638189 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:14.638196 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.639351 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.639359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:14.640371 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:14.640383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.641387 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.641391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:14.641557 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.642107 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:14.642123 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:14.644436 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:14.647583 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:14.647586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:14.647589 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:14.647920 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:14.648337 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:14.648342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:14.648345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:14.648509 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:14.649085 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:14.649090 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:14.649344 (+ 254us) client_negotiation.cc:769] Sending connection context
0928 02:10:14.649526 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 02:10:14.650037 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":168}
I20260928 02:10:14.650696 24860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.633777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.1.182.189:39537)
0928 02:10:14.634397 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:14.634401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:14.634420 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:10:14.634458 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:14.634461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:14.634516 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:14.634632 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:14.636113 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.637190 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.638722 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.639542 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.639662 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.644431 (+ 4769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:14.644438 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:14.644443 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:14.644499 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:14.647400 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.648085 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.648111 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.648116 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.648332 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.648783 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.648787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.648789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.649107 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:14.649367 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:14.649805 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 02:10:14.650075 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":346,"threads_started":1}
I20260928 02:10:14.655429 1754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:10:14.656160 24853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d: Bootstrap starting.
I20260928 02:10:14.659341 24853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:14.660430 24853 log.cc:824] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:14.657752 24678 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "974ab0ceedc546bdaee6611a93e28501"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60422
I20260928 02:10:14.663857 24853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d: No bootstrap required, opened a new log
I20260928 02:10:14.671185 24853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974ab0ceedc546bdaee6611a93e28501" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38039 } } peers { permanent_uuid: "0e70313e1ff846be9a0ffafd316f607c" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } } peers { permanent_uuid: "5dfe9438f3a24a04b92778386853bb4d" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } }
I20260928 02:10:14.671541 24853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dfe9438f3a24a04b92778386853bb4d, State: Initialized, Role: FOLLOWER
I20260928 02:10:14.671983 24853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974ab0ceedc546bdaee6611a93e28501" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38039 } } peers { permanent_uuid: "0e70313e1ff846be9a0ffafd316f607c" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } } peers { permanent_uuid: "5dfe9438f3a24a04b92778386853bb4d" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } }
I20260928 02:10:14.672842 24862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974ab0ceedc546bdaee6611a93e28501" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38039 } } peers { permanent_uuid: "0e70313e1ff846be9a0ffafd316f607c" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } } peers { permanent_uuid: "5dfe9438f3a24a04b92778386853bb4d" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } } }
I20260928 02:10:14.673172 24862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:10:14.674132 24853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:10:14.683848 24873 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 02:10:14.683954 24703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } attempt: 4
W20260928 02:10:14.684090 24873 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:10:14.681388 24785 master_runner.cc:186] Error getting master registration for 127.1.182.190:38039: OK, instance_id { permanent_uuid: "974ab0ceedc546bdaee6611a93e28501" instance_seqno: 1790561413538822 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:10:14.693238 24785 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 02:10:14 dist-test-slave-23bp krb5kdc[24622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561413, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.190@KRBTEST.COM for kudu/127.1.182.188@KRBTEST.COM
I20260928 02:10:14.699689 24856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.684694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57340 (local address 127.1.182.188:33211)
0928 02:10:14.684901 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:14.684908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:14.684931 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:10:14.685160 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:14.685166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:14.685249 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:14.685356 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:14.686300 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.687556 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.689440 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.690194 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.690323 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.694140 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:14.694147 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:14.694151 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:14.694188 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:14.697040 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.697691 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.697696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.697700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.697827 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.698197 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.698204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.698207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.698541 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:14.698731 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:14.699137 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 02:10:14.699328 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:10:14.699707 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.684581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:33211 (local address 127.0.0.1:57340)
0928 02:10:14.684856 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.684876 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:14.685004 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:14.685408 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:14.685413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:14.685939 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:14.686161 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:14.686174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.687710 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.687716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:14.689255 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:14.689271 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.690352 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.690358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:14.690550 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.691245 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:14.691268 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:14.693945 (+ 2677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:14.697177 (+ 3232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:14.697185 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:14.697189 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:14.697567 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:14.697939 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:14.697945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:14.697949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:14.698089 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:14.698671 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:14.698677 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:14.698853 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 02:10:14.698994 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 02:10:14.699316 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260928 02:10:14.706082 24703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501: Bootstrap starting.
I20260928 02:10:14.709658 24703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:14.710707 24703 log.cc:824] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:14.714294 24703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501: No bootstrap required, opened a new log
I20260928 02:10:14.720202 24703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974ab0ceedc546bdaee6611a93e28501" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38039 } } peers { permanent_uuid: "0e70313e1ff846be9a0ffafd316f607c" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } } peers { permanent_uuid: "5dfe9438f3a24a04b92778386853bb4d" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } }
I20260928 02:10:14.720577 24703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974ab0ceedc546bdaee6611a93e28501, State: Initialized, Role: FOLLOWER
I20260928 02:10:14.721098 24703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974ab0ceedc546bdaee6611a93e28501" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38039 } } peers { permanent_uuid: "0e70313e1ff846be9a0ffafd316f607c" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } } peers { permanent_uuid: "5dfe9438f3a24a04b92778386853bb4d" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } }
I20260928 02:10:14.722091 24876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974ab0ceedc546bdaee6611a93e28501" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38039 } } peers { permanent_uuid: "0e70313e1ff846be9a0ffafd316f607c" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } } peers { permanent_uuid: "5dfe9438f3a24a04b92778386853bb4d" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } } }
I20260928 02:10:14.722502 24876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:10:14.723433 24703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:10:14.723582 24678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 02:10:14.723803 24678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:14.727598 24678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974ab0ceedc546bdaee6611a93e28501" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38039 } } peers { permanent_uuid: "0e70313e1ff846be9a0ffafd316f607c" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } } peers { permanent_uuid: "5dfe9438f3a24a04b92778386853bb4d" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } }
I20260928 02:10:14.730556 24678 leader_election.cc:288] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [CANDIDATE]: Term 1 election: Requested vote from peers 0e70313e1ff846be9a0ffafd316f607c (127.1.182.189:39537), 5dfe9438f3a24a04b92778386853bb4d (127.1.182.188:33211)
I20260928 02:10:14.730549 24828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "974ab0ceedc546bdaee6611a93e28501" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5dfe9438f3a24a04b92778386853bb4d"
I20260928 02:10:14.731112 24828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:14.734771 24828 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974ab0ceedc546bdaee6611a93e28501 in term 1.
I20260928 02:10:14.735476 24652 leader_election.cc:302] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dfe9438f3a24a04b92778386853bb4d, 974ab0ceedc546bdaee6611a93e28501; no voters:
I20260928 02:10:14.736007 24876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:10:14.736730 24876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [term 1 LEADER]: Becoming Leader. State: Replica: 974ab0ceedc546bdaee6611a93e28501, State: Running, Role: LEADER
I20260928 02:10:14.737345 24876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974ab0ceedc546bdaee6611a93e28501" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38039 } } peers { permanent_uuid: "0e70313e1ff846be9a0ffafd316f607c" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } } peers { permanent_uuid: "5dfe9438f3a24a04b92778386853bb4d" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } }
I20260928 02:10:14.741201 24886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 974ab0ceedc546bdaee6611a93e28501. Latest consensus state: current_term: 1 leader_uuid: "974ab0ceedc546bdaee6611a93e28501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974ab0ceedc546bdaee6611a93e28501" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38039 } } peers { permanent_uuid: "0e70313e1ff846be9a0ffafd316f607c" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } } peers { permanent_uuid: "5dfe9438f3a24a04b92778386853bb4d" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } } }
I20260928 02:10:14.741705 24886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:14.756186 24890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:10:14.766479 24890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260928 02:10:14.778422 24657 tablet.cc:2369] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:10:14.780193 24777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } attempt: 5
Sep 28 02:10:14 dist-test-slave-23bp krb5kdc[24622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561413, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.182.189@KRBTEST.COM for kudu/127.1.182.188@KRBTEST.COM
I20260928 02:10:14.816565 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.780780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.182.188:33211 (local address 127.0.0.1:57354)
0928 02:10:14.780973 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:14.780996 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:14.781106 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:14.788192 (+ 7086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:14.788201 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:14.788926 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:14.789223 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:14.789242 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.803209 (+ 13967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.803215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:14.804747 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:14.804764 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.806011 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.806018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:14.806241 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.806795 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:14.806818 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:14.809676 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:14.813559 (+ 3883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:14.813566 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:14.813571 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:14.814160 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:14.814657 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:14.814662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:14.814667 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:14.814827 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:14.815539 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:14.815544 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:14.815705 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 02:10:14.815854 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 02:10:14.816155 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 02:10:14.816712 24806 tablet.cc:2369] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:10:14.817324 24856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.753171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57346 (local address 127.1.182.188:33211)
0928 02:10:14.753380 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:14.753389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:14.763112 (+ 9723us) server_negotiation.cc:407] Connection header received
0928 02:10:14.763625 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:14.763635 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:14.763728 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:14.763840 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:14.772300 (+ 8460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.773637 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.789330 (+ 15693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.790349 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.790490 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.800705 (+ 10215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:14.800713 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:14.800721 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:14.800777 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:14.805306 (+ 4529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.808806 (+ 3500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.808814 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.808821 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.814087 (+ 5266us) spinlock_profiling.cc:243] Waited 4.33 ms on lock 0x60c0000347b8. stack: 00007f5af052666e 00007f5af0526430 00007f5aef5ad5ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f5af051c0fd 00007f5af051bd1e 00007f5af7033664 00007f5af7050bfe 00007f5af7049179 00007f5af70428a5 00007f5af6fb0ea5 00007f5af6fafeff 00007f5af6fe382f 00007f5af6fe358d
0928 02:10:14.814257 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.814799 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.814804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.814810 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.815277 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:14.815556 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:14.816216 (+ 660us) server_negotiation.cc:299] Negotiation successful
0928 02:10:14.816528 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"spinlock_wait_cycles":9956736}
I20260928 02:10:14.817736 24900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.781141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57354 (local address 127.1.182.188:33211)
0928 02:10:14.787632 (+ 6491us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:14.787642 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:14.787680 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:10:14.787765 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:14.787774 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:14.787887 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:14.789440 (+ 1553us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:14.789815 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.791547 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.804986 (+ 13439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.805798 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.806549 (+ 751us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.809984 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:14.810004 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:14.810009 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:14.810049 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:14.813388 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.814341 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.814345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.814348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.814516 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.815015 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.815020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.815024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.815383 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:14.816948 (+ 1565us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:14.817092 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 02:10:14.817286 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6292,"thread_start_us":7486,"threads_started":1}
I20260928 02:10:14.820559 24859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.752290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60434 (local address 127.1.182.190:38039)
0928 02:10:14.752990 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:14.752999 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:14.763866 (+ 10867us) server_negotiation.cc:407] Connection header received
0928 02:10:14.764356 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:14.764365 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:14.764460 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:14.764552 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:14.767405 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.768734 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.772773 (+ 4039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.773450 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.773552 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.781501 (+ 7949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:14.781518 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:14.781527 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:14.781588 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:14.815716 (+ 34128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.817635 (+ 1919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.817640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.817643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.817799 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.818298 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.818302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.818305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.818714 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:14.819828 (+ 1114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:14.819986 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 02:10:14.820223 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480}
I20260928 02:10:14.823786 24860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:14.761781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56868 (local address 127.1.182.189:39537)
0928 02:10:14.762022 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:14.762029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:14.776198 (+ 14169us) server_negotiation.cc:407] Connection header received
0928 02:10:14.777116 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:14.777125 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:14.777243 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:14.777323 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:14.791853 (+ 14530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.793136 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.798291 (+ 5155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:14.799189 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:14.799316 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:14.809482 (+ 10166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:14.809491 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:14.809498 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:14.809539 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:14.813081 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.821657 (+ 8576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.821664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.821668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.821816 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:14.822255 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:14.822263 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:14.822266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:14.822616 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:14.822855 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:14.823230 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 02:10:14.823454 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:10:14.824858 24777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e70313e1ff846be9a0ffafd316f607c: Bootstrap starting.
I20260928 02:10:14.829022 24777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e70313e1ff846be9a0ffafd316f607c: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:14.831553 24777 log.cc:824] T 00000000000000000000000000000000 P 0e70313e1ff846be9a0ffafd316f607c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:14.836759 24777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e70313e1ff846be9a0ffafd316f607c: No bootstrap required, opened a new log
I20260928 02:10:14.845273 24828 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d [term 1 FOLLOWER]: Refusing update from remote peer 974ab0ceedc546bdaee6611a93e28501: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:10:14.846359 24886 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dfe9438f3a24a04b92778386853bb4d" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:10:14.809397 24636 master_runner.cc:186] Error getting master registration for 127.1.182.189:39537: OK, instance_id { permanent_uuid: "0e70313e1ff846be9a0ffafd316f607c" instance_seqno: 1790561414053927 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:10:14.849109 24636 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:10:14.861917 24777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e70313e1ff846be9a0ffafd316f607c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974ab0ceedc546bdaee6611a93e28501" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38039 } } peers { permanent_uuid: "0e70313e1ff846be9a0ffafd316f607c" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } } peers { permanent_uuid: "5dfe9438f3a24a04b92778386853bb4d" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } }
I20260928 02:10:14.862718 24777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e70313e1ff846be9a0ffafd316f607c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e70313e1ff846be9a0ffafd316f607c, State: Initialized, Role: FOLLOWER
I20260928 02:10:14.863843 24777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e70313e1ff846be9a0ffafd316f607c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974ab0ceedc546bdaee6611a93e28501" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38039 } } peers { permanent_uuid: "0e70313e1ff846be9a0ffafd316f607c" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } } peers { permanent_uuid: "5dfe9438f3a24a04b92778386853bb4d" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } }
I20260928 02:10:14.879055 24752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "974ab0ceedc546bdaee6611a93e28501" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0e70313e1ff846be9a0ffafd316f607c"
I20260928 02:10:14.879649 24752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e70313e1ff846be9a0ffafd316f607c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:14.883497 24752 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e70313e1ff846be9a0ffafd316f607c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974ab0ceedc546bdaee6611a93e28501 in term 1.
I20260928 02:10:14.884315 24751 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0e70313e1ff846be9a0ffafd316f607c [term 1 FOLLOWER]: Refusing update from remote peer 974ab0ceedc546bdaee6611a93e28501: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:10:14.885442 24886 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e70313e1ff846be9a0ffafd316f607c" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:10:14.890990 24777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e70313e1ff846be9a0ffafd316f607c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:10:14.902693 24862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 974ab0ceedc546bdaee6611a93e28501. Latest consensus state: current_term: 1 leader_uuid: "974ab0ceedc546bdaee6611a93e28501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974ab0ceedc546bdaee6611a93e28501" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38039 } } peers { permanent_uuid: "0e70313e1ff846be9a0ffafd316f607c" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } } peers { permanent_uuid: "5dfe9438f3a24a04b92778386853bb4d" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } } }
I20260928 02:10:14.904475 24862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:10:14.907687 24898 mvcc.cc:205] Tried to move back new op lower bound from 7334139555137576960 to 7334139554769600512. Current Snapshot: MvccSnapshot[applied={T|T < 7334139555137576960}]
I20260928 02:10:14.917439 24876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "974ab0ceedc546bdaee6611a93e28501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974ab0ceedc546bdaee6611a93e28501" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38039 } } peers { permanent_uuid: "0e70313e1ff846be9a0ffafd316f607c" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } } peers { permanent_uuid: "5dfe9438f3a24a04b92778386853bb4d" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } } }
I20260928 02:10:14.917834 24876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:14.919778 24890 catalog_manager.cc:1347] Generated new cluster ID: f228f105fff0413ca9e21d06dae64354
I20260928 02:10:14.919899 24890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:10:14.923759 24905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e70313e1ff846be9a0ffafd316f607c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 974ab0ceedc546bdaee6611a93e28501. Latest consensus state: current_term: 1 leader_uuid: "974ab0ceedc546bdaee6611a93e28501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974ab0ceedc546bdaee6611a93e28501" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38039 } } peers { permanent_uuid: "0e70313e1ff846be9a0ffafd316f607c" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } } peers { permanent_uuid: "5dfe9438f3a24a04b92778386853bb4d" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } } }
I20260928 02:10:14.924069 24903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e70313e1ff846be9a0ffafd316f607c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "974ab0ceedc546bdaee6611a93e28501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974ab0ceedc546bdaee6611a93e28501" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38039 } } peers { permanent_uuid: "0e70313e1ff846be9a0ffafd316f607c" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } } peers { permanent_uuid: "5dfe9438f3a24a04b92778386853bb4d" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } } }
I20260928 02:10:14.924465 24903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e70313e1ff846be9a0ffafd316f607c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:10:14.924510 24905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e70313e1ff846be9a0ffafd316f607c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:10:14.927702 24862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "974ab0ceedc546bdaee6611a93e28501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974ab0ceedc546bdaee6611a93e28501" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38039 } } peers { permanent_uuid: "0e70313e1ff846be9a0ffafd316f607c" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } } peers { permanent_uuid: "5dfe9438f3a24a04b92778386853bb4d" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } } }
I20260928 02:10:14.928000 24862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:10:14.965561 24905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e70313e1ff846be9a0ffafd316f607c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "974ab0ceedc546bdaee6611a93e28501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974ab0ceedc546bdaee6611a93e28501" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38039 } } peers { permanent_uuid: "0e70313e1ff846be9a0ffafd316f607c" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } } peers { permanent_uuid: "5dfe9438f3a24a04b92778386853bb4d" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } } }
I20260928 02:10:14.965865 24905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e70313e1ff846be9a0ffafd316f607c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:10:14.966432 24876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "974ab0ceedc546bdaee6611a93e28501" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974ab0ceedc546bdaee6611a93e28501" member_type: VOTER last_known_addr { host: "127.1.182.190" port: 38039 } } peers { permanent_uuid: "0e70313e1ff846be9a0ffafd316f607c" member_type: VOTER last_known_addr { host: "127.1.182.189" port: 39537 } } peers { permanent_uuid: "5dfe9438f3a24a04b92778386853bb4d" member_type: VOTER last_known_addr { host: "127.1.182.188" port: 33211 } } }
I20260928 02:10:14.966769 24876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501 [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:14.982758 24890 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:10:14.984830 24890 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:10:14.992131 24922 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:10:15.019244 24890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 974ab0ceedc546bdaee6611a93e28501: Generated new TSK 0
I20260928 02:10:15.020702 24890 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:10:15.112959 24900 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:15.100905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.1.182.188:33211)
0928 02:10:15.101116 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:15.101124 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:15.110736 (+ 9612us) server_negotiation.cc:407] Connection header received
0928 02:10:15.111087 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:15.111097 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:15.111236 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:15.111320 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:15.112631 (+ 1311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57360: BlockingRecv error: recv got EOF from 127.0.0.1:57360 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 02:10:15.113778 24859 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:15.100753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60444 (local address 127.1.182.190:38039)
0928 02:10:15.100971 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:15.100978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:15.111677 (+ 10699us) server_negotiation.cc:407] Connection header received
0928 02:10:15.111974 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:15.111982 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:15.112116 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:15.112218 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:15.113510 (+ 1292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60444: BlockingRecv error: recv got EOF from 127.0.0.1:60444 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 02:10:15.116297 24860 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:15.111835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.1.182.189:39537)
0928 02:10:15.112058 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:15.112064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:15.113815 (+ 1751us) server_negotiation.cc:407] Connection header received
0928 02:10:15.114333 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:15.114342 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:15.114470 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:15.114559 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:15.115950 (+ 1391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56878: BlockingRecv error: recv got EOF from 127.0.0.1:56878 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 02:10:15.128352 24860 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:15.126048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.1.182.189:39537)
0928 02:10:15.126282 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:15.126292 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:15.126793 (+ 501us) server_negotiation.cc:407] Connection header received
0928 02:10:15.127056 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:15.127063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:15.127244 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:15.127358 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:15.128113 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56886: BlockingRecv error: recv got EOF from 127.0.0.1:56886 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 02:10:15.137066 24860 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:15.134829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56896 (local address 127.1.182.189:39537)
0928 02:10:15.135099 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:15.135107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:15.135593 (+ 486us) server_negotiation.cc:407] Connection header received
0928 02:10:15.135901 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:15.135906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:15.135988 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:15.136096 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:15.136798 (+ 702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56896: BlockingRecv error: recv got EOF from 127.0.0.1:56896 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260928 02:10:15.144183 24860 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:15.142033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56902 (local address 127.1.182.189:39537)
0928 02:10:15.142235 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:15.142244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:15.142761 (+ 517us) server_negotiation.cc:407] Connection header received
0928 02:10:15.143027 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:15.143033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:15.143171 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:15.143277 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:15.143952 (+ 675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56902: BlockingRecv error: recv got EOF from 127.0.0.1:56902 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 02:10:15.153754 24860 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:15.151603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56906 (local address 127.1.182.189:39537)
0928 02:10:15.151824 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:15.151833 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:15.152339 (+ 506us) server_negotiation.cc:407] Connection header received
0928 02:10:15.152583 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:15.152595 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:15.152665 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:15.152771 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:15.153504 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56906: BlockingRecv error: recv got EOF from 127.0.0.1:56906 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260928 02:10:15.167097 24860 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:15.164892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56910 (local address 127.1.182.189:39537)
0928 02:10:15.165135 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:15.165144 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:15.165669 (+ 525us) server_negotiation.cc:407] Connection header received
0928 02:10:15.165900 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:15.165906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:15.166000 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:15.166114 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:15.166846 (+ 732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56910: BlockingRecv error: recv got EOF from 127.0.0.1:56910 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 02:10:15.188789 24860 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:15.186497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.1.182.189:39537)
0928 02:10:15.186743 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:15.186752 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:15.187319 (+ 567us) server_negotiation.cc:407] Connection header received
0928 02:10:15.187559 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:15.187569 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:15.187657 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:15.187785 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:15.188522 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56914: BlockingRecv error: recv got EOF from 127.0.0.1:56914 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 02:10:15.226625 24860 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:15.224370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56918 (local address 127.1.182.189:39537)
0928 02:10:15.224608 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:15.224618 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:15.225190 (+ 572us) server_negotiation.cc:407] Connection header received
0928 02:10:15.225433 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:15.225438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:15.225515 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:15.225629 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:15.226378 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56918: BlockingRecv error: recv got EOF from 127.0.0.1:56918 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 02:10:15.296665 24860 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:15.294384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.1.182.189:39537)
0928 02:10:15.294653 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:15.294663 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:15.295247 (+ 584us) server_negotiation.cc:407] Connection header received
0928 02:10:15.295499 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:15.295505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:15.295577 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:15.295688 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:15.296427 (+ 739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56930: BlockingRecv error: recv got EOF from 127.0.0.1:56930 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 02:10:15.429874 24860 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:15.428007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56942 (local address 127.1.182.189:39537)
0928 02:10:15.428303 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:15.428312 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:15.428557 (+ 245us) server_negotiation.cc:407] Connection header received
0928 02:10:15.428732 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:15.428736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:15.428813 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:15.428904 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:15.429645 (+ 741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56942: BlockingRecv error: recv got EOF from 127.0.0.1:56942 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 02:10:15.686062 24873 catalog_manager.cc:1259] Loaded cluster ID: f228f105fff0413ca9e21d06dae64354
I20260928 02:10:15.686209 24873 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d: loading cluster ID for follower catalog manager: success
I20260928 02:10:15.689046 24873 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d: acquiring CA information for follower catalog manager: success
I20260928 02:10:15.691403 24873 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5dfe9438f3a24a04b92778386853bb4d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 02:10:15.692847 24860 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:15.690870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56956 (local address 127.1.182.189:39537)
0928 02:10:15.691087 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:15.691095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:15.691447 (+ 352us) server_negotiation.cc:407] Connection header received
0928 02:10:15.691682 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:15.691687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:15.691763 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:15.691852 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:15.692632 (+ 780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56956: BlockingRecv error: recv got EOF from 127.0.0.1:56956 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:10:15.996842 24922 catalog_manager.cc:1259] Loaded cluster ID: f228f105fff0413ca9e21d06dae64354
I20260928 02:10:15.996980 24922 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0e70313e1ff846be9a0ffafd316f607c: loading cluster ID for follower catalog manager: success
I20260928 02:10:15.999718 24922 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0e70313e1ff846be9a0ffafd316f607c: acquiring CA information for follower catalog manager: success
I20260928 02:10:16.001616 24922 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0e70313e1ff846be9a0ffafd316f607c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:10:16.218039 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:16.209407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.1.182.189:39537)
0928 02:10:16.212378 (+ 2971us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:16.212383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:16.212400 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:10:16.212461 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:16.212468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:16.212633 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:16.212782 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:16.213288 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:16.214112 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:16.215615 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:16.216087 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:16.216304 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:16.216823 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:16.217378 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:16.217595 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:16.217648 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 02:10:16.217732 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2835,"thread_start_us":2630,"threads_started":1}
I20260928 02:10:16.231251 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:16.224072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.1.182.188:33211)
0928 02:10:16.224922 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:16.224930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:16.224953 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:10:16.225082 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:16.225088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:16.225301 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:16.225503 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:16.225871 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:16.227101 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:16.228503 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:16.229192 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:16.229340 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:16.229806 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:16.230565 (+ 759us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:16.230723 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:16.230785 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 02:10:16.230873 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":564,"threads_started":1}
I20260928 02:10:16.255735 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:16.238704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57386 (local address 127.1.182.188:33211)
0928 02:10:16.247310 (+ 8606us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:16.247321 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:16.247353 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:10:16.247453 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:16.247462 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:16.247587 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:16.247721 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:16.248711 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:16.250417 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:16.252847 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:16.253536 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:16.253656 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:16.254418 (+ 762us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:16.254593 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:16.254685 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:16.255337 (+ 652us) server_negotiation.cc:299] Negotiation successful
0928 02:10:16.255428 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8398}
I20260928 02:10:16.261000 25001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:16.238651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.1.182.190:38039)
0928 02:10:16.251031 (+ 12380us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:16.251038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:16.251061 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:10:16.251198 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:16.251204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:16.251447 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:16.252421 (+ 974us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:16.252598 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:16.253773 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:16.257383 (+ 3610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:16.258089 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:16.259114 (+ 1025us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:16.259321 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:16.259836 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:16.260520 (+ 684us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:16.260576 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 02:10:16.260667 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12231,"thread_start_us":12955,"threads_started":1}
I20260928 02:10:16.265146 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:16.247324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.1.182.189:39537)
0928 02:10:16.247535 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:16.247541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:16.256505 (+ 8964us) server_negotiation.cc:407] Connection header received
0928 02:10:16.256811 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:16.256817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:16.256924 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:16.256998 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:16.257917 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:16.259279 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:16.261379 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:16.262098 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:16.262206 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:16.263654 (+ 1448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:16.263937 (+ 283us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:16.264050 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:16.264420 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 02:10:16.264587 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 02:10:16.265990 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 24636
I20260928 02:10:16.267467 24698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:10:16.427986 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 24636
W20260928 02:10:16.448807 24728 connection.cc:535] server connection from 127.0.0.1:56850 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:10:16.449155 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 24710
I20260928 02:10:16.450255 24772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:10:16.558105 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 24710
I20260928 02:10:16.579121 1754 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu pid 24785
I20260928 02:10:16.580332 24848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:10:16.690713 1754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktsFKfV/build/asan/bin/kudu with pid 24785
2026-09-28T02:10:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4967 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9894 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (363926 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST